Agent Installation
Last updated
Last updated
OpenJDK 8, for Agent versions < 23.12.1.
OpenJDK 17, starting with Agent 23.12.1.
Note: For Java 8 installs, make sure that you have installed the OpenJDK 8 update 282 or later, otherwise, the Java JCE will also be needed.
Download the Agent package and unzip it into a directory of your choice:
The following example shows version 2.54.0 being unzipped:
The output will look similar to the following:
The Agent distribution package contains:
Agent Server Package: agent-<VERSION>.jar.
Configuration Helper Package: util/setup-VERSION.jar.
Sample Configuration: agent.yml. For information on the fields see Agent Configuration Fields.
Agent API Swagger Specification: resources/agent-api-public.yaml.
Test Certificates: test certificates for the required service users (resources/certs/agentservice.p12 and resources/certs/ceservice.p12).
Test User Certificate: a test user certificate for calling the Agent API (resources/certs/bot.user1.p12; note that you must create the user first and import its certificate or its signing certificate into the pod)
Test Signing Certificate and Key: a test signing certificate and its private key for generating user certs (resources/certs/int-cert.p12 and resources/certs/int-key.pem)
Shell Scripts:
util/create_cert.sh: a shell script to create a certificate signed by the user-provided signing cert.
util/create_rsa.sh: a shell script to create a public/private key pair.
start.sh: startup script for launching the Agent and setting JVM-specific options. TLS 1.2 is enforced.
stop.sh: stop script. Please note that it will work only if the Agent has been started by using start.sh.
startup.sh: legacy startup script for launching the Agent and setting JVM-specific options. Please use the new start.sh if possible; if you use it, stop.sh script will not work and TLS 1.2 is not enforced
Test public/private key pair: for calling the Agent API (resources/rsa/privatekey.pem and resources/rsa/pubkey.pem). Note that you must create the user first and import their public key into the pod.
Sample Postman collection: contains predefined calls for testing the Agent installation (resources/Postman.json).
jwt-helper: a helper tool to generate a JWT authentication token for the provided username and signed by the provided private RSA key (util/jwt-helper.jar). Run as java -jar jwt-helper.jar -user <username> -key <privatekey.pem>); the output is a short-lived (5 minutes) token which can be used in the payload of the RSA authentication requests.
The JCurl library, a curl replacement with native support for PKCS12 certificates and JSON, is no longer packaged with the Agent. If you relied on it for some of your scripts, you can still find it in the following FINOS repository.
Modify the included configuration template (agent.yml) to match your environment.
The sections podName, url, and certificate are required. All other config properties are optional
The users agentservice and ceservice require their certificates or their signing certificate to be imported into the pod
The Key Manager url is different if the Key Manager service is deployed on-prem or on cloud.
It is possible to encrypt the sensitive fields of the Agent configuration such as passwords or keys. See section "Overview of the Setup Script"
Before disabling deprecated endpoints, it is important to consider:
Disadvantages: • The use of old APIs will not be possible, so apps and bots that rely on them will break. • Code adjustments may be needed for apps and bots in case of other endpoints become deprecated in the future.
Advantages: • You will have the latest API versions with the latest features, performance improvements and fixed bugs.
Please note it is recommended to use start.sh script. With it, this section becomes obsolete.
The Agent supports TLS 1.2 by default but also allows lower versions of TLS protocol to be used. To restrict the Agent to only accept TLS 1.2 and strong cipher suites, set the following values by passing them as command line arguments when starting the Agent:
There are two options for running the agent:
For testing purposes, you can run the Agent on the default port (9443) with a supplied test server certificate and using the system Java truststore java -Dfile.encoding=UTF-8 -jar agent-<VERSION>.jar --agent.config=agent.yml
For production deployments, use a customized version of the start.sh script. You can edit the parameters in the section delimited by Customize From Here / Customize To Here:
To trust a pod certificate that is not in the Java cacerts file, uncomment and edit the following lines:
If you want to customize the default folder for output logs (default folder is ./logs) uncomment and edit the following line:
Make the start.sh executable:
chmod 755 start.sh
Run start.sh:
sudo ./start.sh
After the Agent server starts up, call its health check URL to confirm connectivity from the Agent server to the pod and Key Manager:
https://<agent-server>/agent/v3/health/extended
Here is the response payload of the health check extended endpoint:
Make the stop.sh executable:
chmod 755 stop.sh
The application can be stopped by using stop.sh, that sends a kill signal which will be used to gracefully end the Agent by finishing the ongoing requests but without accepting any new one.
Please note that you can use this stop.sh script only if you have used the start.sh script to run the Agent.
The Agent distribution includes a setup helper script (util/setup-<VERSION>.jar) to create a custom configuration and generate certificates.
The setup script provides interactive options that serve three main purposes:
to generate a usable Agent configuration tailored to the your environment (options [b], [c], [h], and optionally [d]);
to provide helper utilities for facilitating Agent use (options [e], [f], [g]).
to encrypt the sensitive passwords and keys of the Agent configuration to avoid having to store them in clear text (option [i]).
Descriptions of the individual options are provided below.
Launch the script with this command:
java -jar util/setup-<VERSION>.jar
The script displays the main menu:
The following describes the various menu options.
Option [a] (View current configuration) displays the current configuration:
Option [b] (Configure base URL (Symphony URL)) specifies the base URL (Symphony URL) to autogenerate the other URLs critical to the functioning of the Agent (options [b]-[j] under "Configure required properties"):
Option [c] (Configure required properties) allows you to customize the URL guessed on the basis of the previous entry. This also lets the you to set the location of the agentservice user certificate or RSA key (required for the OBO and the ceservice user certificate or RSA key (required for Firehose 2.0). If you don't have those yet, they can be generated by options [e] and [f] in the main menu. Entry [c] also allows you to set a "pod name", used internally by Symphony to identify the pod:
Option [d] (Configure proxies) configures HTTP(s) proxies individually for the various components of the system (Symphony - AKA "Pod", Key Manager, Firehose 2.0). You can also optionally configure proxy authentication (username/password) for each of the components, if used:
Option [e] (Generate user certificate) generates certificates for authenticating service users (/data/agent/certs/agentservice.pem and /data/agent/certs/ceservice.pem). Once generated (using sub option [h]), the output file must be provided in the Agent config (sub options [k] through [p] in the submenu of option [b] (Configure base URL)). Certificate generation requires a signing certificate as described below using sub option [a]:
Sub option [a] (Provide signing certificate): specifies a signing certificate that will be used to sign the generated user certificate. Either the signing certificate or the user certificate needs to be imported to the pod using the Admin Console or Create Company Certificate. If the signing certificate is imported, all users whose certificates are signed by that certificate will be able to authenticate; if only a specific user's certificate is imported, only that user will be able to authenticate:
The following image shows how to import the certificate to the pod, using the Admin Console:
And here you can see how to configure the certificates on the App host: A. Install root certs for the pod in the Trust Store of the Bot. B. Install a Client Type cert in the Key Store of the BOT – obtained from internal PKI CA.
The following commands can be used to install the certificates on the Agent app host machine. The first command is used for adding A (from the diagram above) to the Trust Store and the second command is used for adding B to the Key Store:
Option [f] (Generate RSA keypair) generates an RSA public/private key pair for using RSA authentication RSA Bot Authentication Workflow. This submenu lets you choose filenames for the keys and their encoding (PCKS1 or PCKS8), and writes out the key pair. Once generated, the public keys for each user must be imported to the pod using the Admin Console or Update User V2. Note that RSA authentication requires you to set up each user individually; it doesn't have a concept equivalent to a "signing certificate" for all users. The private key filenames should be set in the Agent config. If both RSA and certificate authentication is configured, RSA takes precedence:Write Configuration
The following image shows how to import the public keys to the pod, using the Admin Console. Note that you have to create a service account for the app:
(Deprecated) Option g (Generate startup script): generates a simple shell startup script for starting the Agent. The startup script sets a number of options configured through Java VM properties (-D...) or Spring runtime properties. You can set the location of your Agent JAR file, your custom Agent config, and additional runtime variables not included in the Agent config file:
The following submenu options are available:
Sub option [c] (Configure server port): the port on which to run the Agent. If not set, defaults to 9443.
Sub options [d] (Configure server certificate file) and [e] (Configure server certificate password): the certificate that the Agent server will present to incoming requests. By default the Agent ships with a Symphony testing certificate, but you should change it to one representing your company to prevent SSL validation errors.
Sub options [h] (Configure truststore file) and [i] (Configure truststore password): a custom truststore for validating SSL certificates presented by the customer's Pod and Key Manager. If not set, the default Java truststore is used.
Sub option [j] (Enforce TLS 1.2): for security reasons only the protocol TLS 1.2 using a strong encryption algorithm is accepted. It is enabled by default and you should not disable it unless your bots do not support TLS 1.2 and you do not want to upgrade them (not recommended).
Sub options [k] (Configure debug endpoints username) and [l] (Configure debug endpoints password): a username and password used to access protected endpoints under /actuator (see Debugging Endpoints below).
Sub option [m] (Configure logs directory): a custom directory for Agent logs.
A generated startup script with all options set looks like this:
Option h (Write agent configuration file) is the final step in the generation of the config. It lets the user write the config as YAML or .properties in the directory of their choice.:
This config must be used to start the Agent:
Note that each property in the generated config is annotated with comments describing its purpose.
Option i(Encrypt a configuration property) allows to encrypt the sensitive passwords and keys of the Agent configuration to avoid having to store them in clear text.
Before encrypting, you must provide the encryption password that will be used for the encryption and then the decryption. This password can be either passed through:
As an environment variable, named AGENT_CONFIG_PASSWORD
As a JVM parameter, named AGENT_CONFIG_PASSWORD
Or will be requested directly through the interactive setup script, if none of the above variables are set.
Select Option a to encrypt a property of the agent configuration.
The following properties of the agent.yaml or properties of the environment/JVM parameters can be encrypted:
Select Option b to encrypt credentials stored in a file.
Once the property has been selected, the script will output the resulting encrypted value with the format ENC(..., ...). Please then set the value of the property or parameter required with this encrypted string of characters.
Please note that as the value of the properties will contain parenthesis'(' (special character), you may need to wrap the property values with double quotes "". See below an example:
The steps above need to be repeated for each property or parameter that requires encryption.
Please note that all the properties need to be encrypted with the same password.
Then the password used for the encryption needs to be provided to the Agent before startup, either using an environment variable or a JVM parameter named AGENT_CONFIG_PASSWORD. See the example below.
At startup, the agent will then retrieve the password and decrypt any of the encrypted properties or credentials.
The Agent server includes a number of Actuator endpoints providing debugging information.
In the cloud setup, the actuator endpoints are disabled by default, and in on-prem setups, it has the health and register endpoints enabled by default.
All actuator endpoints are password-protected and are configured on a separate port, with 10443 as the default value. The port can be overridden by setting the property
--management.server.port=<value> when starting the Agent application.
The security username and password can be set by launching the Agent with:
--spring.security.user.name=admin --spring.security.user.password=password . If not set, the security password is automatically generated by the application.
This basic authentication is the default of standard Spring Security, please refer to https://docs.spring.io/spring-boot/docs/2.7.15/reference/htmlsingle/#web.security for details. As stated in this documentation, it provides HTTP Basic security [...] for the entire application (including actuator endpoints if actuator is on the classpath).
Check the Agent startup logs to find the generated value:
Alternatively, if the management port access is already secured and admin/password auth is not required, this can be disabled by setting the property --management.endpoints.authentication.required=false when starting the Agent application.
The following list shows the available actuator endpoints for the Agent application:
Endpoint name and path
Activated
Description
Method
Activation flag
actuator
/agent/actuator
yes
List of all enabled endpoints.
GET
register
/agent/actuator/register
yes
Agent re-registration.
POST
health
/agent/actuator/health
yes
Displays the health status of your application.
GET
info
/agent/v1/info
/agent/actuator/info
no
Displays information about your application.
GET
--management.endpoint.info.enabled=true
env
/agent/actuator/env
no
Displays current environment properties.
GET
--management.endpoint.env.enabled=true
threaddump
/agent/actuator/threaddump
no
Performs a thread dump.
GET
--management.endpoint.threaddump.enabled=true
heapdump
/agent/actuator/heapdump
no
Returns a GZip compressed JVM heap dump.
GET
--management.endpoint.heapdump.enabled=true
metrics
/agent/actuator/metrics
no
Real-time metrics.
GET
--management.endpoint.metrics.enabled=true
prometheus
/agent/actuator/prometheus
no
Prometheus scrapping endpoint.
GET
--management.endpoint.prometheus.enabled=true
Note:
Paths above are relative to the management URL: https://agent-server.symphony.com:10443
The configuration of the agent logging can be done either using the command line parameters (e.g -DlogLevel=DEBUG for example), or directly through the provided log4j2.xml file.
Note: If you are using a custom start script instead of the provided ones (start.sh and startup.sh), please include the location to the log4j2.xml file by adding -Dlogging.config=log4j2.xml in the Java command line in your custom script.
If you want to set DEBUG log level agent.log, set
Note that the above "-D" option must come before "-jar" argument.
Logging Rotation
By default, the Agent has a logging rotation strategy to keep up to 10 local files, where each file can have a max file size of 100 MB. When a file "rotates", it will be automatically zipped and renamed, adding a number at the end. This number corresponds to how many times the file has already rotated. The numbers will stop being added when the file reaches the limit size.
Note: the Agent has 3 log files: agent.log, agent-error.log, and agent-metrics.log.
Example:
Logging rotation can be customized for all 3 files by setting the following options:
With the above, all 3 Agent log files will rotate (independently) when one of them hits 500 MB in "unzipped" size, and will keep up to 20 files, always keeping the latest ones and discarding the older ones if it goes beyond the file limit.
To customize these parameters for each log file independently, the following configuration can be used (it also demonstrates the syntax for sizes):
agent.log:
agent-error.log:
agent-metrics.log:
The parameters presented above can be combined in any configuration, with the more specific parameters taking precedence over the general one. The following example uses all parameters presented until now, but note that the first 2, the more general ones, will be useless in this configuration because all others are being specified:
The previous command will have the same effect as the command below:
Note: Keep in mind that none of the previous configurations need to be set, this is just for customization purposes.
There is no upgrade path for the Agent. The new Agent can be installed on the same Agent Server as any previous version.
Make sure to shut down your agent first, and then keep your configuration and startup/stop script when you upgrade to a new version.
agent.limits.datafeed.queueSize – allows for an override of the datafeed message queue from the default of 250 events unread
Debug logging – turn on Debug level logging by adding this before the “-jar” command
java -Dloglevel=DEBUG -jar agent.jar --agent.config=agent.yml …
If you encounter the following error message in the agent.log file /tmp/libsymphony-crypto-jni-.so: failed to map segment from shared object: Operation not permitted while running the HealthCheck: Add the following to Java flags in startup.sh before the “-jar” to resolve it: -Djava.io.tmpdir=/data/agent{version}/tmp
