Basic Hasura-based API as a service

[rufuspollock]

Epic here #1

When developing data-api I want to be able to create and run integration tests of the app so that I can check how it's working together with postgres and hasura.

Acceptance

• There is a docker-compose environment with the necessary cervices
• There is ci running tests and publishing to a Docker repository

Tasks

Local developer docker-compose

• add services to docker-compose
  • postres (latest official postgres image)
  • hasura (custom image with adding sample data)
• write a basic smoke test checking the system parts are working
  • Find the framework to use (Supertest remains the default for node http server tests ...)

CI

• Dockerfile (FROM nodejs ... + copy files)
• Publish Docker image (used https://github.com/datopian/data-api/new/master?filename=.github%2Fworkflows%2Fdocker-publish.yml&workflow_template=docker-publish for how to)
• Add another docker-compose for ci
  • add data-api image building from source (include tests)
• check code correctness in CI on every push:
  • run prettier
  • run tests
• create new version Docker tags automatically - CI on every push to master:
  • update version (if tests ok)
  • publish new version of the image
  • publish :latest tag with the new changes

Inbox

• Have Hasura based read API
• Dockerize this service and auto publish on merge to master
  • Work out where we publish to (is it dockerhub or is it github packages? (Ask on tech help?)
  • Publish there with continuous deployment (on master)
• Testing
  • Have basic test for this (using cypress (?))
  • CI setup and passing

[EvgeniiaVak]

@rufuspollock I don't think we really need to store hasura settings anywhere or publish a custom hasura image, because what we will set up in hasura is

• mark which tables we want to track (hasura metadata)
• run any sql scripts preparing tables/views (hasura migrations)

That is very project specific, I don't think we can have anything Datopian wide here. We can just launch hasura service in a project (like we would launch postgres):

Example Hasura Dockerfile

FROM hasura/graphql-engine:latest.cli-migrations-v2

COPY migrations /hasura-migrations/
COPY metadata /hasura-metadata/

Example docker-compose containing Postgres, Hasura, A third service

version: '3'
services:
  wrapper:
    image: my-app-wrapper
    build:
      context: wrapper/
      dockerfile: Dockerfile

  graphql-engine:
    image: my-app-hasura
    build:
      context: hasura/
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
    depends_on:
      - db
    restart: always
    environment:
      HASURA_GRAPHQL_DATABASE_URL: postgres://ckan_default:password@db:5432/datastore_default
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      HASURA_GRAPHQL_DEV_MODE: "true"
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
      ## uncomment next line to set an admin secret
      # HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
  db:
    image: my-app-postgres
    build:
      context: postgres/
      dockerfile: Dockerfile
    ports:
      - "5434:5432"
    environment:
      POSTGRES_DB: datastore_default
      POSTGRES_USER: ckan_default
      POSTGRES_PASSWORD: password

[rufuspollock]

Agreed

…

On Wed, 9 Sep 2020, 16:20 EvgeniiaVak, ***@***.***> wrote: @rufuspollock <https://github.com/rufuspollock> I don't think we really need to store hasura settings anywhere or publish a custom hasura image, because what we will set up in hasura is - mark which tables we want to track (hasura metadata <https://hasura.io/docs/1.0/graphql/core/migrations/manage-metadata.html#manage-hasura-metadata> ) - run any sql scripts preparing tables/views (hasura migrations <https://hasura.io/docs/1.0/graphql/core/migrations/migrations-setup.html#migrations-setup> ) That is very project specific, I don't think we can have anything Datopian wide here. We can just launch hasura service in a project (like we would launch postgres): Example Hasura Dockerfile FROM hasura/graphql-engine:latest.cli-migrations-v2 COPY migrations /hasura-migrations/COPY metadata /hasura-metadata/ Example docker-compose containing Postgres, Hasura, A third service version: '3'services: wrapper: image: my-app-wrapper build: context: wrapper/ dockerfile: Dockerfile graphql-engine: image: my-app-hasura build: context: hasura/ dockerfile: Dockerfile ports: - "8080:8080" depends_on: - db restart: always environment: HASURA_GRAPHQL_DATABASE_URL: ***@***.***:5432/datastore_default ## enable the console served by server HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console ## enable debugging mode. It is recommended to disable this in production HASURA_GRAPHQL_DEV_MODE: "true" HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log ## uncomment next line to set an admin secret # HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey db: image: my-app-postgres build: context: postgres/ dockerfile: Dockerfile ports: - "5434:5432" environment: POSTGRES_DB: datastore_default POSTGRES_USER: ckan_default POSTGRES_PASSWORD: password — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABMDMWEFVM3ZZIG3ZP24STSE6FLJANCNFSM4Q767G5A> .

[EvgeniiaVak]

WON'T FIX. See above comments.

[rufuspollock]

@EvgeniiaVak what i would be doing here is listing the docker compose setup you have described in e.g. the README and actually setting that up for this project with CI - so we have a basic test for running hasura against a sample DB (i know this may duplicate some work elsewhere but doing it here should be trivial and is useful IMO).

[EvgeniiaVak]

Tests are passing in docker runner:

https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/07692fac-c5db-407d-ace1-2c6ea400244b

...
$ mocha -r dotenv/config --recursive
data-api
�[0mGET /non_existing_page �[33m404 �[0m4.439 ms - 10�[0m
✓ should return 404 for non existing page (40ms)
GraphQL endpoint
�[0mPOST /v1/graphql �[32m200 �[0m25.855 ms - -�[0m
✓ returns graphql schema (61ms)
2 passing (112ms)
...

[EvgeniiaVak]

Prettier check is running on new builds:

https://hub.docker.com/repository/registry-1.docker.io/datopian/data-api/builds/3ca791c7-b76c-4461-a814-2afa40045a43

Step 9/10 : RUN ["yarn", "prettier", "--check", "."]
---> Running in 79016bfb81d4
yarn run v1.22.4
$ /usr/src/app/node_modules/.bin/prettier --check .
Checking formatting...
All matched files use Prettier code style!

[rufuspollock]

@EvgeniiaVak great - can we tick acceptance then and close?

[EvgeniiaVak]

@rufuspollock let's not close it just yet, I really want to make this automatic version tagging work (right now the it's always data-api:latest), also github seems to run tests too, and they fail (I think additionally to dockerhub (which pass))