mod-reporting

mod-reporting

Copyright (C) 2023-2025 The Open Library Foundation

This software is distributed under the terms of the Apache License, Version 2.0. See the file "LICENSE" for more information.

• Overview
• Compilation and installation
• Configuration
  • Configuration file
  • Logging
  • FOLIO services and reporting databases
• Notes
  • Redundant field in API
  • CORS problems when running locally
• See also
• Additional information
  • Issue tracker
  • Code of Conduct
  • ModuleDescriptor
  • API documentation
  • Code analysis
  • Download and configuration
• Author

Overview

mod-reporting is a FOLIO module that mediates access to reporting databases -- instances of either MetaDB or LDP Classic. It removes the need to deal directly with a relational database by providing a simple WSAPI that can be used by UI code such as ui-ldp.

mod-reporting started out as a plug-compatible replacement for mod-ldp, using the same API specification (FOLIO module descriptor, RAML file and associated JSON Schemas and examples). It provides the same interface (ldp-query) with the same semantics. In more recent releases, it adds new endpoints that were never supported by mod-ldp, but remains strictly backwards compatible.

As well as the machine-readable API specification, a human-readable overview is provided.

Personal data. This module does not store any personal data. See the file PERSONAL_DATA_DISCLOSURE.md for details.

Contributing. See the file CONTRIBUTING.md for details.

Compilation and installation

mod-reporting is written in Go.

Compilation is controlled by a good old-fashioned Makefile

• make target/ModuleDescriptor.json to compile the module descriptor
• make target/mod-reporting to build to module
• make test to run tests
• make lint to audit code quality and security

Invocation takes a single argument, the name of a configuration file (see below). Its behaviour can also be affected by environment variables (see below).

Containerization is supported:

• docker build -t mod-reporting . to create a container
• docker run -p 12369:12369 mod-reporting to run the container with its default port wired out to the host

GitHub workflows exist to build the code and run tests, and to create a Docker container.

Configuration

Configuration file

The configuration file is written in JSON. An example can be found in etc/config.json:

{
  "logging": {
    "categories": "config,listen,path,error",
    "prefix": "",
    "timestamp": false
  },
  "listen": {
    "host": "0.0.0.0",
    "port": 12369
  },
  "queryTimeout": 120,
  "reportUrlWhitelist": [
    "^https://gitlab.com/MikeTaylor/metadb-queries/",
    "^https://raw.githubusercontent.com/metadb-project/"
  ]
}

Four top-level entries are supported:

• logging specifies how the system's categorical logger should be configured:
  • categories is a comma-separated list of logging categories for which output should be emitted: see below
  • prefix is an optional string which will be emitted at the start of each logging line. This can help to differentiate logging output from other outputs.
  • timestamp is a boolean indicating whether each logged line should be timestamped.
• listen specifies where the running server should listen for connections:
  • host is an IP address or DNS-resolvable hostname. 0.0.0.0 (all interfaces) should usually be used
  • port is an IP port number
• queryTimeout specifies how long, in seconds, mod-reporting should allow Postgres to run any query. Running longer than this will result in a timeout error. Defaults to 60 seconds if not specified. See also MOD_REPORTING_QUERY_TIMEOUT below.
• reportUrlWhitelist is an optional list of regular expressions. If this is specified, then only report URLs that match one of these regular expressions are accepted. Note. In the sample configuration file, the whitelist is disabled: for deployments that want to apply this filtering, it is the responsibility of their administrators to modify their configuration accordingly.

The port specified in the listen stanza can be overridden at run-time by setting the SERVER_PORT environment variable. This is useful when invoking the service from a container whose contents (i.e. the configuration file) cannot easily be modified, but whose environment can be specified.

The timeout length specified by the queryTimeout entry in the configuration file can be overridden at run-time by setting the MOD_REPORTING_QUERY_TIMEOUT environment variable.

Logging

The following categories of logging information may be emitted, depending on how the logger is configured:

• config -- logs the contents of the configuration file
• listen -- indicates when the server has started listening, and on what host and port
• path -- notes each path requested by a client
• db -- emits information about each reporting database and notes when successful connections are made
• sql -- logs the generated SQL for each JSON query submitted via the /ldp/db/query endpoint
• validate -- logs checks of report URLs against the specified whitelist regular expressions
• error -- emits error messages returned to the client in HTTP responses

Access to the FOLIO database is performed using the foliogo client library which also uses categorical logger. See its documentation for information on the categories service, session, op, auth, curl, status and response.

NOTE. The curl logging category shows complete curl commands that can be run from the command-line, including live authentication tokens. For this reason it should be used only as a debugging aid, and never included when running in production.

FOLIO services and reporting databases

In normal operation, each incoming request is serviced by reference to the Okapi instance that sent it. For development, however, it's possible to override this behaviour and have every outgoing request go to a nominated Okapi instance. This is specified by the environment variables OKAPI_URL (e.g https://folio-snapshot-okapi.dev.folio.org) and OKAPI_TENANT (e.g. diku). When using an Okapi service specified in this way, authentication onto this instance is done using the values specifid by the environment variables OKAPI_USER and OKAPI_PW.

Similarly, in normal operation, mod-reporting determines which underlying reporting database to connect to on the basis of the information configured in FOLIO -- specifically, in the mod-settings record with scope ui-ldp.config and key dbinfo, as managed by the "Database configuration" settings page of the Reporting app. However, these configured settings can be overridden if mod-reporting is run with all three of the following environment variables set:

• REPORTING_DB_URL -- The full URL of the PostgreSQL database to connect to, e.g. postgres://localhost:5432/metadb
• REPORTING_DB_USER -- The name of the PostgreSQL user to act as when accessing this database
• REPORTING_DB_PASS -- The password to use for nominated user

Notes

Redundant field in API

In the response from /ldp/db/reports, there is a numeric element totalRecords. Note that this is a count of the number of records included in the records array -- not the total number of hits in the database. (That information is not available from PostgreSQL). The provided field is redundant, and would have been better omitted, but we retain it for backwards compatibility.

CORS problems when running locally

If running mod-reporting locally, you will likely run into CORS problems with Stripes refusing to make GET and POST requests to it because OPTIONS requests don't return the necessary Access-control-allow-origin header. To work around this, you can run a CORS-permissive HTTP proxy such as local-cors-anywhere -- which by default listens on port 8080 -- and access the running mod-reporting at http://localhost:8080/http://localhost:12369.

See also

• The ldp-config-tool utility
• The old mod-ldp for which this is a replacement
• FOLIO's Reporting app, ui-ldp, which uses this module
• The FOLIO Module Acceptance assessment

Additional information

Issue tracker

See project MODREP at the FOLIO issue tracker.

Code of Conduct

Refer to the Wiki FOLIO Code of Conduct.

ModuleDescriptor

See the ModuleDescriptor for the interfaces that this module requires and provides, the permissions, and the additional module metadata.

API documentation

API descriptions:

• RAML and the included Overview.
• JSON Schemas

Generated API documentation.

Code analysis

SonarCloud analysis.

Download and configuration

The built artifacts for this module are available. See configuration for repository access, and the Docker images for released versions and for snapshot versions.

Author

Mike Taylor, Index Data ApS. [email protected]