1) you can create a mail server rule on the Domino server, it's in the Domino configuration document -> Router/SMTP tab -> Restrictions and Controls tab -> Rules tab
Create a mail server rule for Size (in bytes) is less than or is greater than any size you want, and you have options to journal this messages, move to database, don't accept message, don't deliver message, change routing state or stop processing.
2) you can also restrict the message size using a setting in the Domino configuration document -> Router/SMTP tab -> Restrictions and controls tab -> Restrictions tab -> "Maximum message size" field.
The maximum message size in KB the server accepts. The router rejects any messages that exceed this size for both transfer and delivery. A non delivery message will be returned to the sender reporting the reason of the failure. The default is 0 KB, which does not limit message size.
Both Size restriction is applied on total size of messages, not attachment. Total size of messages which include the attachment, body of messages and headers information. So if you need to make sure certain size of attachment go through you have to add some additional size for the size restriction on the server.