Use Redis for ActiveJob and ActionCable #2004

2025-03-19T21:16:40+08:00

zachgoll commented

2025-03-19 21:16:40 +08:00

(Migrated from github.com)

Self hosters: BREAKING CHANGES

This PR introduces breaking changes and action is required to keep your self hosted app working and receiving updates

What are the changes?

We will be adding Redis as a required service and moving to Sidekiq for background job processing. This means we will be removing GoodJob so that all apps are running the same job backend for consistency.
We are removing Render as an "officially supported" self hosting method. This will not affect your existing services—it simply means that we as a team will not be "officially" recommending/supporting this hosting method. Our recommendation moving forward is Docker Compose, which we have a setup guide for.

Why the changes?

As we introduce our new AI chat feature and continue to iterate on our background data syncs, we believe that moving away from a database-backed job backend is advantageous for the long-term quality of the app.

How to migrate

Choose the migration guide below based on which hosting platform you are using.

Docker Compose

Below are the steps to migrate your docker compose setup to use the new Maybe stack.

How this impacts you

Background jobs (e.g. "account syncs") will stop working until you make required changes
No data will be altered

Required migration steps

Step 1: Backup your database

While the following changes should not affect your database, it is always good to backup your database when making significant changes to a Docker compose setup.

You can make a quick backup locally by:

Make sure your Maybe app is running
Go to the directory your compose.yml file is in
Run the command: docker compose exec postgres pg_dump -U maybe_user maybe_production > backup_$(date +%Y%m%d).sql where postgres represents the name of the database service in your compose.yml file.

Step 2: Stop your service

docker compose down

Step 3: Replace your compose.yml file with the new example file

The new compose.example.yml file has been refactored to use the new services and should work with your existing .env file. You can simply copy/paste the new file found here:

https://github.com/maybe-finance/maybe/blob/main/compose.example.yml

Step 3: Start your service

As noted in the official guide, you can pull the latest updates and start the service per usual:

docker compose pull # This pulls the "latest" published image from GHCR
docker compose build # This rebuilds the app with updates
docker compose up --no-deps -d

If at any point you get stuck, you can always reference our Docker guide, which takes you step-by-step to getting the Maybe app running:

https://github.com/maybe-finance/maybe/blob/main/docs/hosting/docker.md

Render

Note: your services will not be deleted or altered by this change

How this impacts you

The "Render Blueprint" will no longer sync, and you will be responsible for managing the Render services required to run the Maybe app (Postgres DB, Redis, Web service, Workers)
- We are removing this blueprint because with these additional resources, it is no longer viable to run this for "free" on Render
The "upgrades" module will no longer be present in the application. You should set "auto deploy" to true in your Render dashboard to get automatic updates

Required migration steps

Add a key value store service on Render with a maxmemory policy of noeviction (recommended for queues/persistence)
Add a background worker service to Render and copy over all the environment variables from your main "Web" service (i.e. the Rails app)
Find the internal Redis URL from your key value service, copy it, and set REDIS_URL in both the main Rails "web" service and the new background worker you just created

If you get stuck at any point, Render has an excellent guide explaining how to setup Rails, Sidekiq, and Redis together. The Maybe app's setup is very "standard" and this guide is highly applicable:

https://render.com/docs/deploy-rails-sidekiq

## Self hosters: BREAKING CHANGES **This PR introduces _breaking changes_ and _action is required_ to keep your self hosted app working and receiving updates** ### What are the changes? 1. **We will be adding Redis as a required service** and moving to Sidekiq for background job processing. This means we will be removing GoodJob so that all apps are running the same job backend for consistency. 2. **We are removing Render as an "officially supported" self hosting method.** This will not affect your existing services—it simply means that we as a team will not be "officially" recommending/supporting this hosting method. Our recommendation moving forward is Docker Compose, which we have a [setup guide for](https://github.com/maybe-finance/maybe/blob/main/docs/hosting/docker.md). ### Why the changes? As we introduce our new AI chat feature and continue to iterate on our background data syncs, we believe that moving away from a database-backed job backend is advantageous for the long-term quality of the app. ## How to migrate Choose the migration guide below based on which hosting platform you are using. ### Docker Compose Below are the steps to migrate your docker compose setup to use the new Maybe stack. #### How this impacts you - Background jobs (e.g. "account syncs") will stop working until you make required changes - No data will be altered #### Required migration steps **Step 1: Backup your database** While the following changes should not affect your database, it is always good to backup your database when making significant changes to a Docker compose setup. You can make a quick backup locally by: 1. Make sure your Maybe app is running 2. Go to the directory your `compose.yml` file is in 4. Run the command: `docker compose exec postgres pg_dump -U maybe_user maybe_production > backup_$(date +%Y%m%d).sql` where `postgres` represents the name of the database service in your `compose.yml` file. **Step 2: Stop your service** ``` docker compose down ``` **Step 3: Replace your `compose.yml` file with the new example file** The new `compose.example.yml` file has been refactored to use the new services and should work with your existing `.env` file. You can simply copy/paste the new file found here: https://github.com/maybe-finance/maybe/blob/main/compose.example.yml **Step 3: Start your service** As noted in the [official guide](https://github.com/maybe-finance/maybe/blob/main/docs/hosting/docker.md), you can pull the latest updates and start the service per usual: ``` docker compose pull # This pulls the "latest" published image from GHCR docker compose build # This rebuilds the app with updates docker compose up --no-deps -d ``` If at any point you get stuck, you can always reference our Docker guide, which takes you step-by-step to getting the Maybe app running: https://github.com/maybe-finance/maybe/blob/main/docs/hosting/docker.md ### Render _**Note:** your services will not be deleted or altered by this change_ #### How this impacts you - The "Render Blueprint" will no longer sync, and you will be responsible for managing the Render services required to run the Maybe app (Postgres DB, Redis, Web service, Workers) - We are removing this blueprint because with these additional resources, it is no longer viable to run this for "free" on Render - The "upgrades" module will no longer be present in the application. You should set "auto deploy" to true in your Render dashboard to get automatic updates #### Required migration steps 1. Add a [key value store](https://render.com/docs/key-value) service on Render with a maxmemory policy of `noeviction (recommended for queues/persistence)` 5. Add a [background worker](https://render.com/docs/background-workers) service to Render and copy over all the environment variables from your main "Web" service (i.e. the Rails app) 6. Find the internal Redis URL from your key value service, copy it, and set `REDIS_URL` in both the main Rails "web" service and the new background worker you just created If you get stuck at any point, Render has an excellent guide explaining how to setup Rails, Sidekiq, and Redis together. The Maybe app's setup is very "standard" and this guide is highly applicable: https://render.com/docs/deploy-rails-sidekiq

zachgoll (Migrated from github.com) reviewed 2025-03-20 00:22:48 +08:00

compose.example.yml

						
				@@ -0,0 +34,4 @@

				x-rails-env: &rails_env

				  <<: *db_env

				  SECRET_KEY_BASE: ${SECRET_KEY_BASE:-a7523c3d0ae56415046ad8abae168d71074a79534a7062258f8d1d51ac2f76d3c3bc86d86b6b0b307df30d9a6a90a2066a3fa9e67c5e6f374dbd7dd4e0778e13}

zachgoll (Migrated from github.com) commented

2025-03-20 00:22:48 +08:00

Purely for example purposes (to make the compose setup work without any config). Users should set this in their environment.

al-lac commented

2025-04-14 21:10:35 +08:00

(Migrated from github.com)

@zachgoll could you please mention this in the release notes somewhere?

Some note about breaking changes for self hosters would be good to have there. Otherwise people will break their instances when updating.

@zachgoll could you please mention this in the release notes somewhere? Some note about breaking changes for self hosters would be good to have there. Otherwise people will break their instances when updating.

zachgoll commented

2025-04-14 21:12:12 +08:00

(Migrated from github.com)

@al-lac good call, will get a note added.

🎉 1

ansonryan commented

2025-04-25 11:56:29 +08:00

(Migrated from github.com)

I am using the following docker compose and still have synchronization problems.
Do I need to add other parameters? Currently I set maybe in unraid

services:
maybe:
image: ghcr.io/maybe-finance/maybe:latest
container_name: Maybe
network_mode: bridge
volumes:
- /mnt/user/appdata/maybe/data:/rails/storage
ports:
- 3000:3000
restart: unless-stopped
environment:
SECRET_KEY_BASE:
DB_HOST: IP
DB_PORT: 5432
POSTGRES_USER:
POSTGRES_PASSWORD:
POSTGRES_DB: maybe_production
OPENAI_ACCESS_TOKEN:
REDIS_URL: redis://IP:6379/8
RAILS_FORCE_SSL: true
RAILS_ASSUME_SSL: true
SELF_HOSTED: true
APP_DOMAIN: https://maybe.domain.com
labels:
net.unraid.docker.webui: http://[IP]:[PORT:3000]
net.unraid.docker.icon: https://raw.githubusercontent.com/nwithan8/unraid_templates/master/images/maybe-icon.png
traefik.enable: true
traefik.http.routers.pfm.rule: Host(maybe.domain.com)

I am using the following docker compose and still have synchronization problems. Do I need to add other parameters? Currently I set maybe in unraid services: maybe: image: ghcr.io/maybe-finance/maybe:latest container_name: Maybe network_mode: bridge volumes: - /mnt/user/appdata/maybe/data:/rails/storage ports: - 3000:3000 restart: unless-stopped environment: SECRET_KEY_BASE: DB_HOST: IP DB_PORT: 5432 POSTGRES_USER: POSTGRES_PASSWORD: POSTGRES_DB: maybe_production OPENAI_ACCESS_TOKEN: REDIS_URL: redis://IP:6379/8 RAILS_FORCE_SSL: true RAILS_ASSUME_SSL: true SELF_HOSTED: true APP_DOMAIN: https://maybe.domain.com labels: net.unraid.docker.webui: http://[IP]:[PORT:3000] net.unraid.docker.icon: https://raw.githubusercontent.com/nwithan8/unraid_templates/master/images/maybe-icon.png traefik.enable: true traefik.http.routers.pfm.rule: Host(`maybe.domain.com`)

candideu commented

2025-05-06 11:02:28 +08:00

(Migrated from github.com)

Are there any Railway users who have managed to get this upgrade working?

I'm using this template, and have no idea what I should add/remove: https://railway.com/template/_mFlP4

It's been stuck on "Syncing accounts data" for the last two days:

Are there any Railway users who have managed to get this upgrade working? I'm using this template, and have no idea what I should add/remove: https://railway.com/template/_mFlP4 It's been stuck on "Syncing accounts data" for the last two days: ![image](https://github.com/user-attachments/assets/1b76b146-bdc1-4efd-afb1-43d04fd775ac)

👀 2

Joelute commented

2025-05-07 06:56:26 +08:00

(Migrated from github.com)

Hey @candideu! I've made an updated template for Railway here: https://railway.com/template/dFsiOp?referralCode=vOnDKx. Feel free to check it out :)

mittalyashu (Migrated from github.com) reviewed 2025-05-08 01:08:22 +08:00

compose.example.yml

						
				@@ -0,0 +34,4 @@

				x-rails-env: &rails_env

				  <<: *db_env

				  SECRET_KEY_BASE: ${SECRET_KEY_BASE:-a7523c3d0ae56415046ad8abae168d71074a79534a7062258f8d1d51ac2f76d3c3bc86d86b6b0b307df30d9a6a90a2066a3fa9e67c5e6f374dbd7dd4e0778e13}

mittalyashu (Migrated from github.com) commented

2025-05-08 01:08:22 +08:00

Does the SECRET_KEY_BASE environment shared between the web and Sidekiq service?
Or both the service needs to have separate (unique) secret keys.

Does the `SECRET_KEY_BASE` environment shared between the `web` and `Sidekiq` service? Or both the service needs to have separate (unique) secret keys.

candideu commented

2025-05-17 02:26:22 +08:00

(Migrated from github.com)

Hey @candideu! I've made an updated template for Railway here: https://railway.com/template/dFsiOp?referralCode=vOnDKx. Feel free to check it out :)

Hi @Joelute : Unfortunately, the Maybe service crashed in Railway when I tried to deploy your template...

> Hey @candideu! I've made an updated template for Railway here: https://railway.com/template/dFsiOp?referralCode=vOnDKx. Feel free to check it out :) Hi @Joelute : Unfortunately, the Maybe service crashed in Railway when I tried to deploy your template...

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: gavin/maybe#2004