Discord is a great service, but I've always been a little uncomfortable about having so much of my private online communications being stored on their servers. Maybe I've just been watching too much Louis Rossman lately, but it really seems like tech companies are getting too greedy with all the telemetry, targetted ads, and personal data collection. So the basis of this project is really just to see if I can build myself an alternative, and reduce my exposure to that sort of stuff just a little bit more.
The goal is not actually a complete discord clone. What I have in mind is something small, self-hosted, that I can distribute amongst a small group of friends.
MVP features I'm shooting for include messaging with support for emojis, GIFs, and video files, and reactions (much like discord), just a single group 'channel' to start, and hopefully voice chat.
I'm going with a react front-end, Node/Express backend, and MongoDB for the server. I've been wanting to re-educate myself on the react workflow, as I haven't really touched it much in the past few years, and this seems like a great opportunity to do so. Another reason for the choice of react is that it might be nice to port this to an android app, and for a hobby project like this, react native could be an easy way to do that. So those are our weapons of choice, let's discuss what I've got so far!
I'm starting off with MongoDB Atlas. I may decide to self-host the database later on down the road, but this is a nice way to hit the ground running. Plus, Atlas' free tier is configured as a 'replica set' which is required to use the all important changestream function (more on that later).
So to kick off this project, I set up a free shared cluster, Set up a 'Messages' database, and created a 'home' collection within that. This collection will act as the database for my initial 'channel'. a future goal would be to be able to set up multiple 'channels' just like discord, but for now I'm just getting the blueprint down, so a single collection will work. Messages will be stored with the following format
{ "_id": { "$oid": "661f29cfe6ff8a7ccbbf3c55" }, "content": "this is a test message", "timestamp": "2024-04-17T01:45:51.763Z", "author": "seanl" }
The nice thing about MongoDB is that there's no defined schema, so it's great for projects like this where I'm just kind of throwing things together as I go. Later on I can add a field for reactions, different types of media, and so on. But that's the basic blueprint
So far I've got two routes, '/api/sendmessage', and '/api/messagestream'.
For sending messages, super basic. The BodyParse line enables express to accept JSON, which gets sent off to my collection with the insertOne function
app.use(bodyParser.json()); app.post('/api/sendmessage', (req, res) => { const date = new Date().toISOString(); msgToServer(req.body); const message = req.body res.json(message); }); async function msgToServer(msg){ const client = new MongoClient(uri); await client.connect(); const dbName = "Messages"; const collectionName = "messages_home"; const database = client.db(dbName); const collection = database.collection(collectionName); try { const cursor = await collection.insertOne(msg); } catch (err) { console.error(`Something went wrong trying to find the documents: ${err}\n`); } }
The message streaming was a little bit of a doozy for me, but I think I've settled on a good prototype. Basically, I was looking for a way to maintain an open connection with the MongoDB collection, so that the express server could get real time updates as messages hit the server. This can be accomplished via the changeStream watcher that I mentioned earlier. This does exactly what I need for real time messaging. It sets up a persistent connection with the MongoDB collection, and returns updates in, well, real time. Super cool and not too bad to set up.
The tricky part for me was figuring out how to send that data back to my react front-end. After doing some googling, I settled on the server sent events API to accomplish this. This may turn out to be the wrong choice, but we'll just have to wait and see!
This turned out to be a bit finnicky. I'm sure that was largely due to my rusty react skills, but I do have a few notes that may help with the configuration. From what I gathered, there are a few key things to include in the route configuration in order to get this to work. First, you need to set the status to 200 in order to keep the connection open (citation needed). Then, set your headers exactly as they are below. Note the content-type, connection, and content-encoding in particular. Finally, the res.write() needs to have that data: string prepended to ensure that the event listener on the client side will trigger.
Setting the res.write inside the changeStream works like a charm. This way, any time the changeStream recieves an update from the collection, the express route and take it and send it back to the client.
app.get('/api/messagestream', async (req, res) =>{ res.status(200) .set({ "Content-Type" : "text/event-stream" , "Cache-Control" : "no-cache" , "Connection" : "keep-alive" , "Content-Encoding" : "none" , "Access-Control-Allow-Origin": "*" }) const client = new MongoClient(uri, {useUnifiedTopology: true}); await client.connect(); const dbName = "Messages"; const collectionName = "messages_home"; const database = client.db(dbName); const collection = database.collection(collectionName); const changeStream = collection.watch(); let msg = ''; changeStream.on('change', next => { console.log(next); console.log('changed'); res.write(`data: ${JSON.stringify(next)} \n\n `); }); });
To make use of these express routes, I've set up two components - MessageBox and MessageStream. MessageBox is pretty boring, just a basic form that sends a post request to the sendmessage route. But I would like to show how I completed the server sent event setup so that my react front-end could actually recieve the updates from the changeStream.
I set up an empty state variable with useState. Then inside useEffect, I created my EventSource listener and passed in the messagestream route. This creates a persistent connection to the /api/messagestream route discussed above. eventSource.onmessage is the listener. So whenever a message is recieved, it updates the State array with the new value. UseEffect has a dependency on that array, so whenever it updates, it runs the useEffect hook again and waits for the next input.
One thing to note here, It's important to close the connection even after the eventListener fires successfully. I believe this is because SSE has a maximum number of conections (I believe six). So if you don't close these properly, it will stop creating new ones, and you won't get any new data being passed.
export default function MessageStream() { const [msgStream, setMsgStream] = useState([]); // console.table(msgStream); useEffect( () => { const eventSource = new EventSource("/api/messagestream"); eventSource.onopen = () => console.log("connection opened"); eventSource.onerror = (e) => { console.log("error: ", e); eventSource.close(); console.log("conection closed"); }; eventSource.onmessage = function (event) { try { console.log(JSON.parse(event.data)); console.log(msgStream); setMsgStream([...msgStream, JSON.parse(event.data)]); eventSource.close() } catch (e) { console.log("oof ", e); eventSource.close() } }; }, [msgStream] ); return ( ... map over the msgEffect array, display all messages
Pretty cool! I'm pretty happy with what I've got so far. Whether it will actually work with multiple users is another question, but for now I think I've got a nice little blueprint. Excited to keep going, and I will be providing updates as I go!