Agent Installation
Symphony API Agent
Prerequisites
OpenJDK 8, for Agent versions < 23.12.1.
OpenJDK 17, starting with Agent 23.12.1.
Note: For Java 8 installs, make sure that you have installed the OpenJDK 8 update 282 or later, otherwise, the Java JCE will also be needed.
Agent Download
Download the Agent package and unzip it into a directory of your choice:
Agent DownloadThe following example shows version 2.54.0 being unzipped:
The output will look similar to the following:
The Agent distribution package contains:
Agent Server Package:
agent-<VERSION>.jar
.Configuration Helper Package:
util/setup-VERSION.jar
.Sample Configuration:
agent.yml
. For information on the fields see Agent Configuration Fields.Agent API Swagger Specification:
resources/agent-api-public.yaml
.Test Certificates: test certificates for the required service users (
resources/certs/agentservice.p12
andresources/certs/ceservice.p12
).Test User Certificate: a test user certificate for calling the Agent API (
resources/certs/bot.user1.p12
; note that you must create the user first and import its certificate or its signing certificate into the pod)Test Signing Certificate and Key: a test signing certificate and its private key for generating user certs (
resources/certs/int-cert.p12
andresources/certs/int-key.pem
)Shell Scripts:
util/create_cert.sh
: a shell script to create a certificate signed by the user-provided signing cert.util/create_rsa.sh
: a shell script to create a public/private key pair.start.sh
: startup script for launching the Agent and setting JVM-specific options. TLS 1.2 is enforced.stop.sh
: stop script. Please note that it will work only if the Agent has been started by using start.sh.startup.sh
: legacy startup script for launching the Agent and setting JVM-specific options. Please use the new start.sh if possible; if you use it, stop.sh script will not work and TLS 1.2 is not enforcedTest public/private key pair: for calling the Agent API (
resources/rsa/privatekey.pem
andresources/rsa/pubkey.pem
). Note that you must create the user first and import their public key into the pod.Sample Postman collection: contains predefined calls for testing the Agent installation (
resources/Postman.json
).jwt-helper: a helper tool to generate a JWT authentication token for the provided username and signed by the provided private RSA key (
util/jwt-helper.jar
). Run asjava -jar jwt-helper.jar -user <username> -key <privatekey.pem>
); the output is a short-lived (5 minutes) token which can be used in the payload of the RSA authentication requests.
The JCurl library, a curl replacement with native support for PKCS12 certificates and JSON, is no longer packaged with the Agent. If you relied on it for some of your scripts, you can still find it in the following FINOS repository.
Configure
Modify the included configuration template (agent.yml
) to match your environment.
Important
The sections
podName
,url
, andcertificate
are required. All other config properties are optionalThe users
agentservice
andceservice
require their certificates or their signing certificate to be imported into the podThe Key Manager url is different if the Key Manager service is deployed on-prem or on cloud.
It is possible to encrypt the sensitive fields of the Agent configuration such as passwords or keys. See section "Overview of the Setup Script"
Disabling deprecated endpoints
Before disabling deprecated endpoints, it is important to consider:
Disadvantages: • The use of old APIs will not be possible, so apps and bots that rely on them will break. • Code adjustments may be needed for apps and bots in case of other endpoints become deprecated in the future.
Advantages: • You will have the latest API versions with the latest features, performance improvements and fixed bugs.
Configuring TLS 1.2 (only applies if not using start.sh script)
Please note it is recommended to use start.sh script. With it, this section becomes obsolete.
The Agent supports TLS 1.2 by default but also allows lower versions of TLS protocol to be used. To restrict the Agent to only accept TLS 1.2 and strong cipher suites, set the following values by passing them as command line arguments when starting the Agent:
Run
Start
There are two options for running the agent:
For testing purposes, you can run the Agent on the default port (9443) with a supplied test server certificate and using the system Java truststore
java -Dfile.encoding=UTF-8 -jar agent-<VERSION>.jar --agent.config=agent.yml
For production deployments, use a customized version of the start.sh script. You can edit the parameters in the section delimited by
Customize From Here
/Customize To Here:
To trust a pod certificate that is not in the Java cacerts
file, uncomment and edit the following lines:
If you want to customize the default folder for output logs (default folder is ./logs
) uncomment and edit the following line:
Make the start.sh
executable:
chmod 755 start.sh
Run start.sh
:
sudo ./start.sh
After the Agent server starts up, call its health check URL to confirm connectivity from the Agent server to the pod and Key Manager:
https://<agent-server>/agent/v3/health/extended
Here is the response payload of the health check extended endpoint:
Stop
Make the stop.sh
executable:
chmod 755 stop.sh
The application can be stopped by using stop.sh, that sends a kill signal which will be used to gracefully end the Agent by finishing the ongoing requests but without accepting any new one.
Please note that you can use this stop.sh script only if you have used the start.sh script to run the Agent.
Overview of the Setup Script
The Agent distribution includes a setup helper script (util/setup-<VERSION>.jar
) to create a custom configuration and generate certificates.
The setup script provides interactive options that serve three main purposes:
to generate a usable Agent configuration tailored to the your environment (options
[b]
,[c]
,[h]
, and optionally[d]
);to provide helper utilities for facilitating Agent use (options
[e]
,[f]
,[g]
).to encrypt the sensitive passwords and keys of the Agent configuration to avoid having to store them in clear text (option
[i]
).
Descriptions of the individual options are provided below.
Launch the Setup Script
Launch the script with this command:
java -jar util/setup-<VERSION>.jar
The script displays the main menu:
The following describes the various menu options.
View current configuration
Option [a]
(View current configuration
) displays the current configuration:
Configure base URL
Option [b]
(Configure base URL (Symphony URL)
) specifies the base URL (Symphony URL) to autogenerate the other URLs critical to the functioning of the Agent (options [b]-[j] under "Configure required properties"):
Configure required properties
Option [c]
(Configure required properties
) allows you to customize the URL guessed on the basis of the previous entry. This also lets the you to set the location of the agentservice user certificate or RSA key (required for the OBO and the ceservice user certificate or RSA key (required for Firehose 2.0). If you don't have those yet, they can be generated by options [e]
and [f]
in the main menu. Entry [c]
also allows you to set a "pod name", used internally by Symphony to identify the pod:
Configure proxies
Option [d]
(Configure proxies
) configures HTTP(s) proxies individually for the various components of the system (Symphony - AKA "Pod", Key Manager, Firehose 2.0). You can also optionally configure proxy authentication (username/password) for each of the components, if used:
Generate user certificate
Option [e]
(Generate user certificate
) generates certificates for authenticating service users (/data/agent/certs/agentservice.pem
and /data/agent/certs/ceservice.pem
). Once generated (using sub option [h]
), the output file must be provided in the Agent config (sub options [k]
through [p]
in the submenu of option [b]
(Configure base URL
)). Certificate generation requires a signing certificate as described below using sub option [a]
:
Sub option
[a]
(Provide signing certificate
): specifies a signing certificate that will be used to sign the generated user certificate. Either the signing certificate or the user certificate needs to be imported to the pod using the Admin Console or Create Company Certificate. If the signing certificate is imported, all users whose certificates are signed by that certificate will be able to authenticate; if only a specific user's certificate is imported, only that user will be able to authenticate:
The following image shows how to import the certificate to the pod, using the Admin Console:
And here you can see how to configure the certificates on the App host: A. Install root certs for the pod in the Trust Store of the Bot. B. Install a Client Type cert in the Key Store of the BOT – obtained from internal PKI CA.
The following commands can be used to install the certificates on the Agent app host machine. The first command is used for adding A (from the diagram above) to the Trust Store and the second command is used for adding B to the Key Store:
Generate RSA keypair
Option [f]
(Generate RSA keypair
) generates an RSA public/private key pair for using RSA authentication RSA Bot Authentication Workflow. This submenu lets you choose filenames for the keys and their encoding (PCKS1 or PCKS8), and writes out the key pair. Once generated, the public keys for each user must be imported to the pod using the Admin Console or Update User V2. Note that RSA authentication requires you to set up each user individually; it doesn't have a concept equivalent to a "signing certificate" for all users. The private key filenames should be set in the Agent config. If both RSA and certificate authentication is configured, RSA takes precedence:Write Configuration
The following image shows how to import the public keys to the pod, using the Admin Console. Note that you have to create a service account for the app:
Generate startup script (deprecated)
(Deprecated) Option g
(Generate startup script
): generates a simple shell startup script for starting the Agent. The startup script sets a number of options configured through Java VM properties (-D...) or Spring runtime properties. You can set the location of your Agent JAR file, your custom Agent config, and additional runtime variables not included in the Agent config file:
The following submenu options are available:
Sub option
[c]
(Configure server port
): the port on which to run the Agent. If not set, defaults to 9443.Sub options
[d]
(Configure server certificate file
) and[e]
(Configure server certificate password
): the certificate that the Agent server will present to incoming requests. By default the Agent ships with a Symphony testing certificate, but you should change it to one representing your company to prevent SSL validation errors.Sub options
[h]
(Configure truststore file
) and[i]
(Configure truststore password
): a custom truststore for validating SSL certificates presented by the customer's Pod and Key Manager. If not set, the default Java truststore is used.Sub option
[j]
(Enforce TLS 1.2): for security reasons only the protocol TLS 1.2 using a strong encryption algorithm is accepted. It is enabled by default and you should not disable it unless your bots do not support TLS 1.2 and you do not want to upgrade them (not recommended).Sub options
[k]
(Configure debug endpoints username
) and[l]
(Configure debug endpoints password
): a username and password used to access protected endpoints under /actuator (see Debugging Endpoints below).Sub option
[m]
(Configure logs directory
): a custom directory for Agent logs.
A generated startup script with all options set looks like this:
Write agent configuration file
Option h
(Write agent configuration file
) is the final step in the generation of the config. It lets the user write the config as YAML or .properties in the directory of their choice.:
This config must be used to start the Agent:
Note that each property in the generated config is annotated with comments describing its purpose.
Encrypt a configuration property
Option i
(Encrypt a configuration property
) allows to encrypt the sensitive passwords and keys of the Agent configuration to avoid having to store them in clear text.
Before encrypting, you must provide the encryption password that will be used for the encryption and then the decryption. This password can be either passed through:
As an environment variable, named AGENT_CONFIG_PASSWORD
As a JVM parameter, named AGENT_CONFIG_PASSWORD
Or will be requested directly through the interactive setup script, if none of the above variables are set.
Select Option a
to encrypt a property of the agent configuration.
The following properties of the agent.yaml or properties of the environment/JVM parameters can be encrypted:
Select Option b
to encrypt credentials stored in a file.
Once the property has been selected, the script will output the resulting encrypted value with the format ENC(..., ...). Please then set the value of the property or parameter required with this encrypted string of characters.
Please note that as the value of the properties will contain parenthesis'('
(special character), you may need to wrap the property values with double quotes "". See below an example:
The steps above need to be repeated for each property or parameter that requires encryption.
Please note that all the properties need to be encrypted with the same password.
Then the password used for the encryption needs to be provided to the Agent before startup, either using an environment variable or a JVM parameter named AGENT_CONFIG_PASSWORD.
See the example below.
At startup, the agent will then retrieve the password and decrypt any of the encrypted properties or credentials.
Debugging Endpoints
The Agent server includes a number of Actuator endpoints providing debugging information.
In the cloud setup, the actuator endpoints are disabled by default, and in on-prem setups, it has the health
and register
endpoints enabled by default.
All actuator endpoints are password-protected and are configured on a separate port, with 10443 as the default value. The port can be overridden by setting the property
--management.server.port=<value>
when starting the Agent application.
Security
The security username and password can be set by launching the Agent with:
--spring.security.user.name=admin --spring.security.user.password=password
. If not set, the security password is automatically generated by the application.
This basic authentication is the default of standard Spring Security, please refer to https://docs.spring.io/spring-boot/docs/2.7.15/reference/htmlsingle/#web.security for details. As stated in this documentation, it provides HTTP Basic security [...] for the entire application (including actuator endpoints if actuator is on the classpath).
Check the Agent startup logs to find the generated value:
Alternatively, if the management port access is already secured and admin/password auth is not required, this can be disabled by setting the property --management.endpoints.authentication.required=false
when starting the Agent application.
Endpoints
The following list shows the available actuator endpoints for the Agent application:
Endpoint name and path | Activated | Description | Method | Activation flag |
actuator
| yes | List of all enabled endpoints. | GET | |
register
| yes | Agent re-registration. | POST | |
health
| yes | Displays the health status of your application. | GET | |
info
| no | Displays information about your application. | GET |
|
env
| no | Displays current environment properties. | GET |
|
threaddump
| no | Performs a thread dump. | GET |
|
heapdump
| no | Returns a GZip compressed JVM heap dump. | GET |
|
metrics
| no | Real-time metrics. | GET |
|
prometheus
| no | Prometheus scrapping endpoint. | GET |
|
Note:
Paths above are relative to the management URL: https://agent-server.symphony.com:10443
Logging
Configuration of the logs
The configuration of the agent logging can be done either using the command line parameters (e.g -DlogLevel=DEBUG
for example), or directly through the provided log4j2.xml file.
Note: If you are using a custom start script instead of the provided ones (start.sh and startup.sh), please include the location to the log4j2.xml file by adding -Dlogging.config=log4j2.xml
in the Java command line in your custom script.
Logging Level
If you want to set DEBUG log level agent.log, set
Note that the above "-D" option must come before "-jar" argument.
Logging Rotation
By default, the Agent has a logging rotation strategy to keep up to 10 local files, where each file can have a max file size of 100 MB. When a file "rotates", it will be automatically zipped and renamed, adding a number at the end. This number corresponds to how many times the file has already rotated. The numbers will stop being added when the file reaches the limit size.
Note: the Agent has 3 log files: agent.log, agent-error.log, and agent-metrics.log.
Example:
Logging rotation can be customized for all 3 files by setting the following options:
With the above, all 3 Agent log files will rotate (independently) when one of them hits 500 MB in "unzipped" size, and will keep up to 20 files, always keeping the latest ones and discarding the older ones if it goes beyond the file limit.
To customize these parameters for each log file independently, the following configuration can be used (it also demonstrates the syntax for sizes):
agent.log:
agent-error.log:
agent-metrics.log:
The parameters presented above can be combined in any configuration, with the more specific parameters taking precedence over the general one. The following example uses all parameters presented until now, but note that the first 2, the more general ones, will be useless in this configuration because all others are being specified:
The previous command will have the same effect as the command below:
Note: Keep in mind that none of the previous configurations need to be set, this is just for customization purposes.
Upgrade
There is no upgrade path for the Agent. The new Agent can be installed on the same Agent Server as any previous version.
Make sure to shut down your agent first, and then keep your configuration and startup/stop script when you upgrade to a new version.
Troubleshooting
agent.limits.datafeed.queueSize
– allows for an override of the datafeed message queue from the default of 250 events unreadDebug logging – turn on Debug level logging by adding this before the “-jar” command
java -Dloglevel=DEBUG -jar agent.jar --agent.config=agent.yml …
If you encounter the following error message in the agent.log file
/tmp/libsymphony-crypto-jni-.so: failed to map segment from shared object: Operation not permitted
while running the HealthCheck: Add the following to Java flags in startup.sh before the “-jar” to resolve it:-Djava.io.tmpdir=/data/agent{
version
}/tmp
Last updated