Aggregation query to return most recent messages to and from User A not working properly

I have a MostRecentMessage model, defined as,

const MostRecentMessage = new Schema({
  to: {
    type: mongoose.Schema.Types.ObjectId,
    ref: "user"
  },
  from: {
    type: mongoose.Schema.Types.ObjectId,
    ref: "user"
  },
  conversation: {
    type: mongoose.Schema.Types.ObjectId,
    ref: "conversation"
  },
  date: {
    type: Date,
    default: Date.now
  }
});

Whenever User A sends a message to User B, a document is created if it doesn't exist. Every time User A sends User B a message, this document is updated with the new conversation and date. An example of a documents is,

_id: ObjectId(5dc46521cf670a0017d2434d)
to: ObjectId(5dc464ce2fd75700178c1ad4) // User B
from: ObjectId(5dc464fc2fd75700178c1ad5) // User A
conversation: ObjectId(5dc465c6cf670a0017d24363)
date: 2019-11-07T18:40:33.242+00:00
__v: 0

The idea is that this would help track all recent messages to specific users.

Likewise, User B could send User A a message. A new document would be created, similar to the above,

_id: ObjectId(5dc46521cf670a0017d2434d)
to: ObjectId(5dc464fc2fd75700178c1ad5) // User A
from: ObjectId(5dc464ce2fd75700178c1ad4) // User B
conversation: ObjectId(5dc465c6cf670a0017d24363)
date: 2019-11-07T18:40:33.242+00:00
__v: 0

And this document would be updated with every new message that User B sends to User A.

Now, below is my aggregation query.. but the problem is that it only returns one conversation. There could be 10 Users talking to User A, but it would still return only one document.

  const { id } = req.user;

  try {
    await MostRecentMessages.aggregate(
      [
        {
          $match: {
            $or: [
              { from: mongoose.Types.ObjectId(id) },
              { to: mongoose.Types.ObjectId(id) }
            ]
          }
        },
        { $project: { _id: 1, from: 1, to: 1, conversation: 1, date: 1 } },
        { $sort: { date: -1 } },
        {
          $group: {
            _id: null,
            from: { $first: "$from" },
            to: { $first: "$to" },
            date: { $first: "$date" },
            conversation: { $first: "$conversation" }
          }
        },
        {
          $lookup: {
            from: "conversations",
            localField: "conversation",
            foreignField: "_id",
            as: "conversation"
          }
        },
        { $unwind: { path: "$conversation" } },
        {
          $lookup: {
            from: "users",
            localField: "to",
            foreignField: "_id",
            as: "to"
          }
        },
        { $unwind: { path: "$to" } },
        {
          $lookup: {
            from: "users",
            localField: "from",
            foreignField: "_id",
            as: "from"
          }
        },
        { $unwind: { path: "$from" } }
      ],
      function(err, docs) {
        if (err) {
          console.log(err);
        } else {
          return res.json(docs);
        }
      }
    );
  } catch (err) {
    console.log(err);

    return res.status(500).send("Server error");
  }

What am I doing wrong here?