Data Source: StarRocks

This article will introduce how to deploy Metad Analytics Cloud privately and integrate it with the StarRocks data source to provide powerful multidimensional data indicator analysis capabilities for enterprises and data analysis teams. In this article, we will use tools such as Docker Compose to build and run this private deployment program.

Introduction

Metad Analytics Cloud is an advanced data analysis platform that helps users get data from a variety of data sources and perform efficient data exploration and visual analysis. StarRocks is a fast, scalable distributed OLAP database that is particularly suitable for multidimensional data analysis. Integrating these two tools together can enhance data analysis capabilities and improve data insights.

Preparation

Before you start deploying, make sure you have installed Docker and Docker Compose. You can download and install these tools from the official website, and make sure they are running properly on your system.

Source Code

The source code of this article can be found on GitHub: https://github.com/meta-d/installer/

Build StarRocks Docker Image

If your company already has its own StarRocks cluster, you can skip this step. If not, you can build StarRocks docker image and deploy with MetaAnalysis cloud program, so that you can test and demonstrate.

The Dockerfile file used to build the StarRocks docker image can be found in the docker/starrocks folder of this article's source code. This StarRocks image runs the fe and be services in the same container, which is convenient for running and testing. It is not recommended to do this in a production environment.

Docker Compose File

Next, we will write a Docker Compose file to define the entire integrated deployment environment. Create a file named docker-compose-with-starrocks.yml in the project docker directory (this file is the content in the source code), and use the following content:

docker-compose-with-starrocks.yml

version: '3'

networks:
  overlay:
    driver: bridge
  starrocks_net:
    ipam:
      config:
        - subnet: 172.20.80.0/16
volumes:
  postgres_data:

services:
  starrocks:
    build:
      context: ./starrocks
    container_name: "metad-starrocks"
    hostname: "fe"
    ports:
      - 8030:8030
      - 9030:9030
      - 8040:8040
    volumes:
      - ./starrocks/fe/meta:/opt/apache-doris/fe/meta
      - ./starrocks/fe/log:/data/deploy/starrocks/fe/log
      - ./starrocks/be/storage:/data/deploy/starrocks/be/storage
      - ./starrocks/be/log:/data/deploy/starrocks/be/log
      - ./initdb.d/:/docker-entrypoint-initdb.d/
    networks:
      - starrocks_net
  ...

The file starts a StarRocks service to simulate a formal cluster environment. Other Metad Analytics Cloud Services are not listed. Please refer to the installation configuration of the Standard Edition.

Standard Edition Deployment

The deployment file of Metad Analytics Cloud standard version only contains the container service of Metad Analytics Cloud itself, not the third-party database or data warehouse service such as StarRocks. This allows users to connect their own StarRocks clusters for data analysis after starting Metad Analytics Cloud.

For the deployment of the standard version of Metad Analytics Cloud, please refer to installation and deployment of docker cluster.

docker-compose.yml

version: '3'
name: 'metad-analytics-cloud'

networks:
  overlay:
    driver: bridge
volumes:
  postgres_data:

services:
  db:
    image: postgres:12-alpine
    container_name: metad-db
    restart: always
    environment:
      POSTGRES_DB: ${DB_NAME:-postgres}
      POSTGRES_USER: ${DB_USER:-postgres}
      POSTGRES_PASSWORD: ${DB_PASS:-root}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "psql postgres://$${POSTGRES_USER}:$${POSTGRES_PASSWORD}@localhost:5432/$${POSTGRES_DB} || exit 1",
        ]
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./initdb.d/:/docker-entrypoint-initdb.d/:ro
    networks:
      - overlay
  redis:
    image: redis:6-alpine
    container_name: metad-redis
    mem_limit: 100m
    restart: unless-stopped
    command: ["sh", "-c", "redis-server --requirepass $${REDIS_PASSWORD}"]
    environment:
      REDIS_PASSWORD: ${REDIS_PASSWORD:-}
    networks:
      - overlay
  olap:
    image: metadc/metad-olap:1.0.0
    container_name: metad-olap
    restart: unless-stopped
    healthcheck:
      test: curl -m 5 --silent --fail --request GET http://localhost:8080/ | jq --exit-status -n 'inputs | if has("status") then .status=="UP" else false end' > /dev/null || exit 1
      interval: 10s
      timeout: 2s
      retries: 10
    links:
      - db:${DB_HOST:-db}
    environment:
      OLAP_REDIS_DATABASE: 1
      OLAP_REDIS_HOST: "redis"
      OLAP_REDIS_PORT: 6379
      OLAP_REDIS_PASSWORD: ${REDIS_PASSWORD:-}
    networks:
      - overlay
  api:
    image: registry.cn-hangzhou.aliyuncs.com/metad/metad-api:1.5.0
    container_name: metad-api
    environment:
      INSTALLATION_MODE: ${INSTALLATION_MODE:-standalone}
      HOST: ${API_HOST:-api}
      PORT: 3000
      NODE_ENV: ${NODE_ENV:-development}
      DB_HOST: db
      REDIS_HOST: redis
      REDIS_PORT: 6379
      OLAP_HOST: olap
      OLAP_PORT: 8080
      API_BASE_URL: ${API_BASE_URL:-http://localhost:3000}
      DEMO: ${DEMO:-false}
      STARROCKS_VERSION: ${}
      STARROCKS_HOST: ${STARROCKS_HOST}
      STARROCKS_PORT: ${STARROCKS_PORT}
      STARROCKS_USER: ${STARROCKS_USER}
      STARROCKS_PASS: ${STARROCKS_PASS}
      STARROCKS_API_PORT: ${STARROCKS_API_PORT}
      SENTRY_DSN: ${SENTRY_DSN:-}
      LOGGER_LEVEL: ${LOGGER_LEVEL:-info}
    env_file:
      - .env
    command: ["node", "main.js"]
    restart: on-failure
    depends_on:
      - db
      - redis
    links:
      - db:${DB_HOST:-db}
    ports:
      - ${API_PORT:-3000}:3000
    volumes:
      - ./public:/srv/pangolin/apps/
    networks:
      - overlay
  webapp:
    image: registry.cn-hangzhou.aliyuncs.com/metad/metad-webapp:1.5.0
    container_name: metad-webapp
    environment:
      HOST: ${WEB_HOST:-webapp}
      PORT: 4200
      NODE_ENV: ${NODE_ENV:-development}
      API_BASE_URL: ${API_BASE_URL:-http://localhost:3000}
      CLIENT_BASE_URL: ${CLIENT_BASE_URL:-http://localhost:4200}
      SENTRY_DSN: ${SENTRY_DSN:-}
      DEFAULT_LATITUDE: ${DEFAULT_LATITUDE:-42.6459136}
      DEFAULT_LONGITUDE: ${DEFAULT_LONGITUDE:-23.3332736}
      DEFAULT_CURRENCY: ${DEFAULT_CURRENCY:-USD}
      API_HOST: ${API_HOST:-api}
      API_PORT: ${API_PORT:-3000}
    env_file:
      - .env
    restart: on-failure
    links:
      - api:${API_HOST:-api}
    depends_on:
      - api
    ports:
      - ${WEBAPP_PORT:-80}:4200
    networks:
      - overlay

The following environment variables need to be specially configured (other variables are configured according to the standard version):

• INSTALLATION_MODE=with-starrocks Configure the installation mode as with-starrocks, so that the metad analytics cloud will create a StarRocks data source and import data when creating demo data.
• DEMO=true Configure as demo mode, so that the metad analystics cloud will import demo data when creating demo data.
• StarRocks connection information related environment variables:
  • STARROCKS_VERSION=2.5
  • STARROCKS_HOST=
  • STARROCKS_PORT=9030
  • STARROCKS_USER=
  • STARROCKS_PASS=

Configure the StarRocks connection information to the connection information of your StarRocks cluster.

Run the Standard Version

Run Docker setup command：

docker-compose up -d

This will start the metad analytics cloud service according to the definition in the Compose file and initialize the system data so that you can log in to the system.

Generate Demo Data

Access metad analytics cloud through the http://localhost/ address or your own domain. Use the administrator account local.admin@mtda.cloud to log in. Once logged in, on the homepage, click on 🔘Generate Demo Samples to generate demonstration data. The demo data includes creating StarRocks data sources, importing data, and creating metrics and storytelling dashboards.

Within metad analytics cloud, you can perform multidimensional data analysis by creating different dashboards, charts, and queries. Leveraging the rich features and intuitive interface of metad analytics cloud, you can easily explore, filter, and visualize data from the StarRocks data sources.

Summary

Through the use of Docker Compose and containerization technology, we have successfully integrated the metad analytics cloud and StarRocks data sources, bringing more powerful multi-dimensional data indicator analysis capabilities to data analysis. Such integration can help companies and data teams better understand data, make wise decisions, and get deeper insights from data. At the same time, the Dockerized deployment method simplifies the integration process, making deployment and management more flexible and efficient.