h3n3/cadhub
1
0
Fork 0
Code Issues 35 Pull Requests Packages Projects Releases Wiki Activity

Add s3 integration #317

Merged
Irev-Dev merged 1 commits from kurt/s3-integration-316-rebase into main 2021-05-17 23:19:48 +02:00
Conversation 2 Commits 1 Files Changed 17 +4766 -980
Irev-Dev commented 2021-05-16 12:35:21 +02:00 (Migrated from github.com)
Copy Link

Preview link

Doing so has a number of benefits

  • Overcome the 10Mb limit of the API gateway the lambdas have to go
    through
  • By storing the key as the hash of the code we can return previous
    generated assets, i.e. caching
  • cost, transfering assets into the bucket within the AWS ecosystem
    is faster than return, and there fore the lambdas execute for less time
  • Sets us up for the future as when generating artifacts for repos when
    there is a change to master etc we want to store these assets somewhere
    and s3 is an obvious choice
  • Solved a weird CORS issue where I couldn't get CORS working with
    binaryMediaTypes enabled, don't need binary types when dumping in s3

Resolves #316

Example of it working with a large mesh (~40Mb)
image

## [Preview link](https://60a0f9f7c1a56fac0b75dc6d--cadhubxyz.netlify.app/) Doing so has a number of benefits - Overcome the 10Mb limit of the API gateway the lambdas have to go through - By storing the key as the hash of the code we can return previous generated assets, i.e. caching - cost, transfering assets into the bucket within the AWS ecosystem is faster than return, and there fore the lambdas execute for less time - Sets us up for the future as when generating artifacts for repos when there is a change to master etc we want to store these assets somewhere and s3 is an obvious choice - Solved a weird CORS issue where I couldn't get CORS working with binaryMediaTypes enabled, don't need binary types when dumping in s3 Resolves #316 Example of it working with a large mesh (~40Mb) ![image](https://user-images.githubusercontent.com/29681384/118394475-4f77f080-b688-11eb-938d-e021e2a93d74.png)
Irev-Dev (Migrated from github.com) reviewed 2021-05-16 12:48:33 +02:00
app/api/src/docker/cadquery/cadquery.js
@@ -38,0 +22,4 @@
const params = {
Bucket: process.env.BUCKET,
Key: key,
}
Irev-Dev (Migrated from github.com) commented 2021-05-16 12:36:49 +02:00
Copy Link

This was me handling CORS preflight endpoints directly because I couldn't get the config working. Turns out it had something to do with binary media types which we don't need since we're not returning the asset directly now.

This was me handling CORS preflight endpoints directly because I couldn't get the config working. Turns out it had something to do with binary media types which we don't need since we're not returning the asset directly now.
app/api/src/docker/cadquery/cadquery.js
Irev-Dev (Migrated from github.com) commented 2021-05-16 12:38:52 +02:00
Copy Link

Return early if the asset is already in the bucket. (based on the key which is a hash of the user's code)

Return early if the asset is already in the bucket. (based on the key which is a hash of the user's code)
app/api/src/docker/common/utils.js
@@ -4,0 +4,4 @@
const { createHash } = require('crypto')
const CONSOLE_MESSAGE_KEY = 'console-message-b64'
function putConsoleMessageInMetadata(consoleMessage) {
Irev-Dev (Migrated from github.com) commented 2021-05-16 12:41:16 +02:00
Copy Link

I'm storing whatever the cad package printed to the terminal when it made the part in metadata, so that I can still display this when returning the cached assets. s3 seems to be pretty sensitive about special characters that can be put in s3 so base64 to the rescue.

I'm storing whatever the cad package printed to the terminal when it made the part in metadata, so that I can still display this when returning the cached assets. s3 seems to be pretty sensitive about special characters that can be put in s3 so base64 to the rescue.
app/api/src/docker/common/utils.js
@@ -38,0 +72,4 @@
})
}
function loggerWrap(handler) {
Irev-Dev (Migrated from github.com) commented 2021-05-16 12:43:06 +02:00
Copy Link

You can create a signed url for a resource that doesn't exist (I guess aws assume you know if it does or not)
so we have to fetch the object's metadata/headObject to see if it exists, and we can grab the console message from metadata while we're at it.

You can create a signed url for a resource that doesn't exist (I guess aws assume you know if it does or not) so we have to fetch the object's metadata/headObject to see if it exists, and we can grab the console message from metadata while we're at it.
app/api/src/docker/common/utils.js
@@ -38,0 +102,4 @@
console.log(`got result in route: ${consoleMessage}, file is: ${fullPath}`)
const { readFile } = require('fs/promises')
let buffer
Irev-Dev (Migrated from github.com) commented 2021-05-16 12:44:10 +02:00
Copy Link

After a part is successful generated the following steps is common to both OpenSCAD and CadQuery so threw it in a function, should probably be cleaned up a little.

After a part is successful generated the following steps is common to both OpenSCAD and CadQuery so threw it in a function, should probably be cleaned up a little.
app/api/src/docker/openscad/openscad.js
Irev-Dev (Migrated from github.com) commented 2021-05-16 12:44:54 +02:00
Copy Link

Should probably just delete this. I should get around to adding stl download soon. Wouldn't be hard.

Should probably just delete this. I should get around to adding stl download soon. Wouldn't be hard.
app/api/src/docker/serverless.yml
@@ -33,4 +28,4 @@
# you can overwrite defaults here
# stage: dev
# region: us-east-1
Irev-Dev (Migrated from github.com) commented 2021-05-16 12:46:04 +02:00
Copy Link

Removing this magically fixed CORS issues I was having as I mentioned in a previous comment.

Removing this magically fixed CORS issues I was having as I mentioned in a previous comment.
app/web/src/components/IdeViewer/IdeViewer.js
@@ -29,3 +37,3 @@
}, [stlData])
}, [url])
if (!loadedGeometry) return null
return (
Irev-Dev (Migrated from github.com) commented 2021-05-16 12:47:29 +02:00
Copy Link

Because the asset is not being returned directly, but instead a URL to it, I now need to make sure the loading animation stays on while the asset loads, especially for the stls as they can be large.

Because the asset is not being returned directly, but instead a URL to it, I now need to make sure the loading animation stays on while the asset loads, especially for the stls as they can be large.
Irev-Dev (Migrated from github.com) reviewed 2021-09-04 10:31:37 +02:00
app/api/src/docker/common/utils.js
@@ -4,0 +4,4 @@
const { createHash } = require('crypto')
const CONSOLE_MESSAGE_KEY = 'console-message-b64'
function putConsoleMessageInMetadata(consoleMessage) {
Irev-Dev (Migrated from github.com) commented 2021-09-04 10:31:37 +02:00
Copy Link

Ended up being a bad idea as only 2kb can be store as metadata, and it's definitely possible for logs to exceed this.

Ended up being a bad idea as only 2kb can be store as metadata, and it's definitely possible for logs to exceed this.
Sign in to join this conversation.
Reviewers
No Reviewers
Irev-Dev
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
h3n3
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: h3n3/cadhub#317
Delete Branch "kurt/s3-integration-316-rebase"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?