Add basic project search #104

New Issue

2020-11-17T07:38:24+01:00

Irev-Dev commented

2020-11-17 07:38:24 +01:00

(Migrated from github.com)

Something basic will do using native database quries will be fine. See the prisma docs
https://www.prisma.io/docs/concepts/components/prisma-client/filtering#filter-conditions

We should be able to use "contains" on the title and description for a very basic search.

We might add elastic search or algolia later for a more advanced implementation

if you want to pick this one up, let me know and I'll try and get you designs soon. But it's basically going to be a search field, then displaying projects with the projects cards that already exist on the home page https://cadhub.xyz/

Something basic will do using native database quries will be fine. See the prisma docs https://www.prisma.io/docs/concepts/components/prisma-client/filtering#filter-conditions We should be able to use "contains" on the title and description for a very basic search. We might add elastic search or algolia later for a more advanced implementation if you want to pick this one up, let me know and I'll try and get you designs soon. But it's basically going to be a search field, then displaying projects with the projects cards that already exist on the home page https://cadhub.xyz/

Isaac-Tait commented

2021-12-10 16:27:29 +01:00

(Migrated from github.com)

Came over after your RedwoodJS talk the other day. Would Algolia be a good solution for this? I set it up on my GatsbyJS site and their free tier works well...

Irev-Dev commented

2021-12-11 04:27:43 +01:00

(Migrated from github.com)

Hey @Isaac-Tait, Oh nice, presenting is paying off 😁

I think Algolia would be perfect.

Just to be clear are you offering to implement this feature? If so should I assign it to you?

Hey @Isaac-Tait, Oh nice, presenting is paying off 😁 I think Algolia would be perfect. Just to be clear are you offering to implement this feature? If so should I assign it to you?

👍 1

Isaac-Tait commented

2021-12-11 16:47:59 +01:00

(Migrated from github.com)

Yes please. I would be happy to be assigned this issue. I am in the midst of implementing Algolia on another project. So, this would be a good project for me to take on.

🎉 1

Irev-Dev commented

2021-12-12 20:38:08 +01:00

(Migrated from github.com)

Awesome, have at it!

I've had a chat to @franknoirot and he's going to put together some designs for the search. It's up to you if you also want to implement the designs. If you're able to get things set up with Agolia and have a basic version working. I'll be happy to match the designs 😁

Contributing.md is the place to look for getting setup, though do let me know if you need help or if anything doesn't make sense.

Awesome, have at it! I've had a chat to @franknoirot and he's going to put together some designs for the search. It's up to you if you also want to implement the designs. If you're able to get things set up with Agolia and have a basic version working. I'll be happy to match the designs 😁 [Contributing.md](https://github.com/Irev-Dev/cadhub/blob/main/CONTRIBUTING.md) is the place to look for getting setup, though do let me know if you need help or if anything doesn't make sense.

🎉 1

Isaac-Tait commented

2021-12-13 18:25:20 +01:00

(Migrated from github.com)

Sounds great. I would be happy to implement the @franknoirot designs. I will get started on this asap 🤩

❤️ 1

franknoirot commented

2021-12-13 20:08:23 +01:00

(Migrated from github.com)

@Isaac-Tait excellent, I'll make some designs and comment them in here after work today. Feel free to tag me with any clarifying questions.

❤️ 2

franknoirot commented

2021-12-14 03:55:19 +01:00

(Migrated from github.com)

@Isaac-Tait I've made just a few designs within this page of our Figma project, and I've added some functional and design notes as comments within there. While you're in the file you can hit the C key to enter comment mode and you should see them as clickable icons over the designs.

After chatting with @Irev-Dev here are some goals for search divided between this V1 implementation and future work:

This iteration
- Fuzzy text search of project titles
- Filtering by CAD package
- URL-driven search (if you visit https://cadhub.xyz/s?q=donut&c=openscad it should load with the search and filter applied)
Future work (just as an FYI and for us to remember)
- Pagination or virtual list
- @ to search for people
- Sorting of results
- Grid/list view toggle

@Isaac-Tait I've made just a few designs within [this page of our Figma project](https://www.figma.com/file/VUh53RdncjZ7NuFYj0RGB9/?node-id=2313%3A5666), and I've added some functional and design notes as comments within there. While you're in the file you can hit the `C` key to enter comment mode and you should see them as clickable icons over the designs. After chatting with @Irev-Dev here are some goals for search divided between this V1 implementation and future work: - This iteration - Fuzzy text search of project titles - Filtering by CAD package - URL-driven search (if you visit https://cadhub.xyz/s?q=donut&c=openscad it should load with the search and filter applied) - Future work (just as an FYI and for us to remember) - Pagination or virtual list - `@` to search for people - Sorting of results - Grid/list view toggle

Irev-Dev commented

2021-12-14 10:42:14 +01:00

(Migrated from github.com)

@Isaac-Tait I only have a crude understand of how solutions like agolia work, from what I understand is you tell it which database columns to index for the search? in which case would the project title and the project description make sense?

Irev-Dev commented

2021-12-14 11:03:36 +01:00

(Migrated from github.com)

I just had a look at the desgins too @franknoirot, they look great and the comments are really good too. Thanks for putting that together.

Frank has done a good job of pointing out where a few components are. Just to expand on something he said about the TopNav component (app\web\src\components\TopNav\TopNav.tsx)

Generate a new page with yarn rw g page search and use the TopNav to make the search bar, that component takes children to go in the center of the bar, so just put the search as a child.

In regards to Franks pattern of cadhub.xyz/s?q=[search-term]&c=[cad-package] using the generate command above will automatically add a new route in web/src/Routes.tsx but with the address of search so just change that to s, and than using a useEffect something like this that runs once when the component mounts.

 const [searchParam, setSearchParam] = useState('')
 const [cadPackage, setCadPackage] = useState('all')
 
 const getSearchParams = async () => {
    const params = new URLSearchParams(window.location.search)
    setSearchParam(params.get('s'))
    setCadPackage(params.get('c') || 'all')

    // do something with search params
  }

  React.useEffect(() => {
    getSearchParams()
  }, [])

Would work for grabbing params on page load so that links with the params in them work (the above hasn't been tested).

I just had a look at the desgins too @franknoirot, they look great and the comments are really good too. Thanks for putting that together. Frank has done a good job of pointing out where a few components are. Just to expand on something he said about the TopNav component (`app\web\src\components\TopNav\TopNav.tsx`) Generate a new page with `yarn rw g page search` and use the TopNav to make the search bar, that component takes children to go in the center of the bar, so just put the search as a child. In regards to Franks pattern of `cadhub.xyz/s?q=[search-term]&c=[cad-package]` using the generate command above will automatically add a new route in `web/src/Routes.tsx` but with the address of `search` so just change that to `s`, and than using a useEffect something like this that runs once when the component mounts. ```javascript const [searchParam, setSearchParam] = useState('') const [cadPackage, setCadPackage] = useState('all') const getSearchParams = async () => { const params = new URLSearchParams(window.location.search) setSearchParam(params.get('s')) setCadPackage(params.get('c') || 'all') // do something with search params } React.useEffect(() => { getSearchParams() }, []) ``` Would work for grabbing params on page load so that links with the params in them work (the above hasn't been tested).

👍 1

Isaac-Tait commented

2021-12-16 15:28:24 +01:00

(Migrated from github.com)

I am running into an error when following the contributing documents. When I run yarn rw prisma migrate dev I get the error:

error Couldn't find a package.json file in "/Users/isaactait/Desktop/Web_Dev"

A similar error cropped up earlier this year in RedwoodJS but was patched... https://github.com/redwoodjs/redwood/issues/222 At this time I am not sure if a similar issue has resurfaced or if something is not right on my end (the most likely culprit 😂).

Can anyone confirm that this is an issue for them as well?

It should be pretty easy to duplicate if you 1️⃣ fork/clone the repo, 2️⃣ cd into it, 3️⃣ run yarn install and 4️⃣ then yarn rw prisma migrate dev

I am running into an error when following the [contributing](https://github.com/Irev-Dev/cadhub/blob/main/CONTRIBUTING.md) documents. When I run `yarn rw prisma migrate dev` I get the error: ``` error Couldn't find a package.json file in "/Users/isaactait/Desktop/Web_Dev" ``` A similar error cropped up earlier this year in RedwoodJS but was patched... https://github.com/redwoodjs/redwood/issues/222 At this time I am not sure if a similar issue has resurfaced or if something is not right on my end (the most likely culprit 😂). Can anyone confirm that this is an issue for them as well? It should be pretty easy to duplicate if you 1️⃣ fork/clone the repo, 2️⃣ cd into it, 3️⃣ run `yarn install` and 4️⃣ then `yarn rw prisma migrate dev`

Irev-Dev commented

2021-12-16 19:10:56 +01:00

(Migrated from github.com)

Did you cd into the app directory @Isaac-Tait?

Yarn install and other commands won't work from the root directory of the repo.

Did you `cd` into the `app` directory @Isaac-Tait? Yarn install and other commands won't work from the root directory of the repo. ![image](https://user-images.githubusercontent.com/29681384/146425878-1c10ceb3-c1ee-4095-9afd-3f3aee67d30f.png)

Irev-Dev commented

2021-12-16 19:21:54 +01:00

(Migrated from github.com)

maybe I should update the docs to cd cadhub/app

maybe I should update the docs to `cd cadhub/app`

Isaac-Tait commented

2021-12-16 19:25:44 +01:00

(Migrated from github.com)

Yup, I was pretty sure it was something I was doing wrong on my end. I understood the docs to mean cd into cadhub not drill down one more level. Once I went into app it went smoothly. Thank you for getting me going in the right direction 🤣 What you have now in the docs makes sense now that you pointed it out. Perhaps updating the docs would make it clearer... or I could have just paid better attention 🤷🏻‍♂️

Yup, I was pretty sure it was something I was doing wrong on my end. I understood the docs to mean `cd into cadhub` not drill down one more level. Once I went into `app` it went smoothly. Thank you for getting me going in the right direction 🤣 What you have now in the docs makes sense now that you pointed it out. Perhaps updating the docs would make it clearer... or I could have just paid better attention 🤷🏻‍♂️

Irev-Dev commented

2021-12-16 19:58:18 +01:00

(Migrated from github.com)

Done 😁

cd3060b3c7

Done 😁 https://github.com/Irev-Dev/cadhub/commit/cd3060b3c729d791f8965c93f939304067dd91b8

🚀 1

Isaac-Tait commented

2021-12-18 16:42:18 +01:00

(Migrated from github.com)

I am figuring this out for the first time as I go and I am stuck on how to proceed through connecting to a local DB.

I am at the point where I need to connect to Postgres. It says I should

Add a DATABASE_URL to your .env file with the URL of the database you'd like to use locally.

If I follow the Redwood "tutorial" and use their database URL I get an error saying I do not have permission to access the database.

If I run psql I see that I have four databases. So, how do I get the URL to one of those databases to use locally? It says that the DB URL must start with the protocol postgresql:// or postgres://

I am figuring this out for the first time as I go and I am stuck on how to proceed through connecting to a local DB. I am at the point where I need to [connect to Postgres](https://redwoodjs.com/docs/local-postgres-setup#connect-to-postgres). It says I should > Add a DATABASE_URL to your .env file with the URL of the database you'd like to use locally. If I follow the Redwood "tutorial" and use _their_ database URL I get an error saying I do not have permission to access the database. If I run `psql` I see that I have four databases. So, how do I get the URL to one of those databases to use locally? It says that the `DB URL must start with the protocol postgresql:// or postgres://`

Isaac-Tait commented

2021-12-18 16:52:36 +01:00

(Migrated from github.com)

Generate a new page with yarn rw g page search and use the TopNav to make the search bar, that component takes children to go in the center of the bar, so just put the search as a child.

Thank you for the explanation. Makes sense, I should be able to do that 😄 The figma designs are very cool, thank you for those @franknoirot

I only have a crude understand of how solutions like agolia work, from what I understand is you tell it which database columns to index for the search? in which case would the project title and the project description make sense?

I can have Algolia populate an index via a GraphQL query. I can also setup multiple indices from two queries too (a feature I am excited to try out as I have read about it but have yet had an opportunity to implement). So, a search for designs and @people would come from two separate indices that are created in the Algolia "backend"

If I am understanding your question then yes searching for a cadhub project title and/or cadhub description would be the best way to go.

> Generate a new page with `yarn rw g page search` and use the TopNav to make the search bar, that component takes children to go in the center of the bar, so just put the search as a child. Thank you for the explanation. Makes sense, I should be able to do that 😄 The figma designs are very cool, thank you for those @franknoirot > I only have a crude understand of how solutions like agolia work, from what I understand is you tell it which database columns to index for the search? in which case would the project title and the project description make sense? I can have Algolia populate an index via a GraphQL query. I can also setup multiple indices from two queries too (a feature I am excited to try out as I have read about it but have yet had an opportunity to implement). So, a search for designs and @people would come from two separate indices that are created in the Algolia "backend" If I am understanding your question then yes searching for a cadhub project title and/or cadhub description would be the best way to go.

Irev-Dev commented

2021-12-19 00:43:04 +01:00

(Migrated from github.com)

@Isaac-Tait you env var should look like the one in the redwood docs DATABASE_URL="postgresql://postgres@localhost:5432/redwoodblog_dev?connection_limit=1" where postgres is the username and redwoodblog_dev is the db name, the latter you can change to a sensible name, maybe cadhub_dev, that becomes the name of the db. postgres will create the db iff it doesn't exist.

If you have permission issues, they are permission issues with postgres itself and the user, not a particular db. It would be useful if you post the error you are seeing but to take a swing. What user are you using?

When I type psql it shows me my user / whoami is kurthutten

Therefore my url is DATABASE_URL="postgresql://kurthutten@localhost:5432/cadhub?connection_limit=1"

If you are still having permission error I think you might need to check the permissions of your user.

What OS are you using?

Hope that helps.

@Isaac-Tait you env var should look like the one in the redwood docs `DATABASE_URL="postgresql://postgres@localhost:5432/redwoodblog_dev?connection_limit=1"` where `postgres` is the username and `redwoodblog_dev` is the db name, the latter you can change to a sensible name, maybe `cadhub_dev`, that becomes the name of the db. postgres will create the db iff it doesn't exist. If you have permission issues, they are permission issues with postgres itself and the user, not a particular db. It would be useful if you post the error you are seeing but to take a swing. What user are you using? When I type `psql` it shows me my user / whoami is `kurthutten` ![image](https://user-images.githubusercontent.com/29681384/146658349-a03d94e2-ebb8-459c-bb3d-c6a25b203871.png) Therefore my url is `DATABASE_URL="postgresql://kurthutten@localhost:5432/cadhub?connection_limit=1"` If you are still having permission error I think you might need to check the permissions of your user. What OS are you using? Hope that helps.

Isaac-Tait commented

2021-12-29 17:18:20 +01:00

(Migrated from github.com)

@Irev-Dev Great, that cleared up the error I had encountered. I am up and running now 🥳

👍 1

Irev-Dev commented

2022-01-07 23:00:25 +01:00

(Migrated from github.com)

Good news! How's it been going?

Isaac-Tait commented

2022-01-11 17:08:16 +01:00

(Migrated from github.com)

Hello @Irev-Dev it has proven to be more in-depth than I was expecting. Gatsby's (where all of my experience with Algolia currently resides) use of GraphQL is very different from RedwoodJS. I have had to learn from the ground up Apollo/Helix (not a bad thing at all - just taking time). Then there is the issue of Redwood having web and api side of things that have to all be tied in with auth, which makes it kinda tricky to get Algolia working on all three fronts. Also there is the issue that my local environment does not have any projects
which will prove difficult to fill the test Algolia indice so I can fine tune the Figma designs and search functionality (that issue is still a ways out from having to be dealt with...).

I have not thrown in the towel yet though. It is just taking me longer than I had originally hoped. 🤣 🤯

Hello @Irev-Dev it has proven to be more in-depth than I was expecting. Gatsby's (where all of my experience with Algolia currently resides) use of GraphQL is very different from RedwoodJS. I have had to learn from the ground up Apollo/Helix (not a bad thing at all - just taking time). Then there is the issue of Redwood having `web` and `api` side of things that have to all be tied in with `auth`, which makes it kinda tricky to get Algolia working on all three fronts. Also there is the issue that my local environment does not have any projects ![Projects](https://user-images.githubusercontent.com/14932877/148977538-c02bfb89-87dd-4e40-b54c-d4762c38366f.png) which will prove difficult to fill the test Algolia indice so I can fine tune the Figma designs and search functionality (that issue is still a ways out from having to be dealt with...). I have not thrown in the towel yet though. It is just taking me longer than I had originally hoped. 🤣 🤯

Irev-Dev commented

2022-01-13 21:53:11 +01:00

(Migrated from github.com)

Sorry I'm a little slow to this.

I appreciate you pushing through this. Interesting the use of GraphQL is different.

How can I help? If there's a chunk of work, or something you'd like me to investigate?

In regards to the you not having any projects locally I can probably figure out solution to that.

Sorry I'm a little slow to this. I appreciate you pushing through this. Interesting the use of GraphQL is different. How can I help? If there's a chunk of work, or something you'd like me to investigate? In regards to the you not having any projects locally I can probably figure out solution to that.

Isaac-Tait commented

2022-01-16 16:29:59 +01:00

(Migrated from github.com)

In regards to the you not having any projects locally I can probably figure out solution to that.

It would be awesome if you could help with that. Thank you 👍🏼 😄

Right now I created a search page. The next step is to sort out the graphQL query that will send the CadHub projects to the Algolia "Design" indices. Algolia will convert that into JSON data that will be used to fulfill searches from the client...

Once we get the search page up and running it should be fairly straightforward in getting the Figma designs up and running.

Gatsby has a pretty cool graphQL interface called GraphiQL. The "Explorer" pane makes it pretty easy to build and test queries before pushing them to your site. It also makes it pretty easy to use GraphQL without knowing much about GraphQL 😆

> In regards to the you not having any projects locally I can probably figure out solution to that. It would be awesome if you could help with that. Thank you 👍🏼 😄 Right now I created a [search page](https://github.com/Isaac-Tait/algolia-cadhub-search-project/blob/0ede72739d53d1ab05f5772841d8a54696133cd5/app/web/src/pages/SearchPage/SearchPage.tsx). The next step is to sort out the graphQL query that will send the CadHub projects to the Algolia "Design" indices. Algolia will convert that into JSON data that will be used to fulfill searches from the client... Once we get the search page up and running it should be fairly straightforward in getting the Figma designs up and running. Gatsby has a pretty cool graphQL interface called [GraphiQL](https://www.gatsbyjs.com/docs/how-to/querying-data/running-queries-with-graphiql/). The "Explorer" pane makes it pretty easy to build and test queries before pushing them to your site. It also makes it pretty easy to use GraphQL without knowing much about GraphQL 😆

Isaac-Tait commented

2022-01-16 16:38:56 +01:00

(Migrated from github.com)

The nice thing about utilizing a GraphQL query to populate the Algolia indices is that every time you build and deploy the site the Projects are sent to Algolia. So, if you add a new project or update the information within a project then build and deploy CadHub Algolia will get the new and/or updated information and instantly provide that to the client during a search.

That does raise a question though. When a user uploads a new project (or edits an existing one) does that trigger a build/deploy within Netlify? If it does not then you would have to build/deploy the site to trigger Algolia to crawl the projects and update the indices. That seems like it would be a huge pain so if that is the case then I might need to look into how to solve that...

The nice thing about utilizing a GraphQL query to populate the Algolia indices is that every time you build and deploy the site the Projects are sent to Algolia. So, if you add a new project or update the information within a project then build and deploy CadHub Algolia will get the new and/or updated information and instantly provide that to the client during a search. That does raise a question though. When a user uploads a new project (or edits an existing one) does that trigger a build/deploy within Netlify? If it does not then you would have to build/deploy the site to trigger Algolia to crawl the projects and update the indices. That seems like it would be a huge pain so if that is the case then I might need to look into how to solve that...

Irev-Dev commented

2022-01-17 10:55:32 +01:00

(Migrated from github.com)

Okay I'll look into getting more projects locally, I'll try and see if I can solve that by the end of the week.

Redwood boots up a graphql playground with the dev process (very similar to graphiql) I think it's at http://localhost:8911/graphql

Short answer to rebuilding and redeploying when users add or change a project. No. We simply can't do that, I heard of people using Gatsby when they shouldn't have for pages with user content and regretted for a similar reason, it required rebuilds all the time and the builds would get slower and slower over time.

I haven't used Agolia before, but have worked adjacent to elastic search, and I know we leveraged a feature that allowed the index it be added to over time with individual data/projects, so initially you do a full index of the site, but after that you give it a single project at a time as users add or change them. Maybe Agolia has a similar feature?

Okay I'll look into getting more projects locally, I'll try and see if I can solve that by the end of the week. Redwood boots up a graphql playground with the dev process (very similar to graphiql) I think it's at http://localhost:8911/graphql Short answer to rebuilding and redeploying when users add or change a project. No. We simply can't do that, I heard of people using Gatsby when they shouldn't have for pages with user content and regretted for a similar reason, it required rebuilds all the time and the builds would get slower and slower over time. I haven't used Agolia before, but have worked adjacent to elastic search, and I know we leveraged a feature that allowed the index it be added to over time with individual data/projects, so initially you do a full index of the site, but after that you give it a single project at a time as users add or change them. Maybe Agolia has a similar feature?

Isaac-Tait commented

2022-01-20 16:56:11 +01:00

(Migrated from github.com)

Short answer to rebuilding and redeploying when users add or change a project. No. We simply can't do that, I heard of people using Gatsby when they shouldn't have for pages with user content and regretted for a similar reason, it required rebuilds all the time and the builds would get slower and slower over time.

That makes total sense and kind of what I was afraid of when I asked 😆

I had an idea in the middle of the night last night. Instead of trying to push the data to Algolia using GraphQL I could just manually upload it. The data would just need to be in JSON, CSV, or TSV format for me to add it to Algolia... Once that is done and we have search functionality we can figure out how to update regularly. This way we are breaking it down into smaller chunks... What format is the db data currently in and are you comfortable sending it to me (Dropbox, GDrive...)?

> Short answer to rebuilding and redeploying when users add or change a project. No. We simply can't do that, I heard of people using Gatsby when they shouldn't have for pages with user content and regretted for a similar reason, it required rebuilds all the time and the builds would get slower and slower over time. That makes total sense and kind of what I was afraid of when I asked 😆 I had an idea in the middle of the night last night. Instead of trying to push the data to Algolia using GraphQL I could just manually upload it. The data would just need to be in JSON, CSV, or TSV format for me to add it to Algolia... Once that is done and we have search functionality we can figure out how to update regularly. This way we are breaking it down into smaller chunks... What format is the db data currently in and are you comfortable sending it to me (Dropbox, GDrive...)?

Irev-Dev commented

2022-01-22 10:39:09 +01:00

(Migrated from github.com)

The db is a postgress db, and I don't think it's a good idea that I send it to you. However I have found a solution for local projects. is isaac@mountaintopcoding.com the best way to contact you @Isaac-Tait?

Isaac-Tait commented

2022-01-24 14:51:47 +01:00

(Migrated from github.com)

@Irev-Dev yes that is my personal email and the best way to get a hold of me. Hope you had a nice weekend 😄

Irev-Dev commented

2022-01-24 20:28:00 +01:00

(Migrated from github.com)

Sweet thanks,

In regards to the index issue does this help https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/in-depth/the-different-synchronization-strategies/#full-record-updates
It sounds like full-record-updates would suit us no? as we know records change, we can update the record as apart of creating or updating project?

Sweet thanks, In regards to the index issue does this help https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/in-depth/the-different-synchronization-strategies/#full-record-updates It sounds like full-record-updates would suit us no? as we know records change, we can update the record as apart of creating or updating project?

👀 1

Isaac-Tait commented

2022-02-11 18:01:23 +01:00

(Migrated from github.com)

I pinged the Redwood forum regarding this project and got some very helpful feedback from Rob and David T. While sorting out how to write a script to populate the Algolia indices I came across Algolia Crawler. It works with the free plan with sites hosted by Netlify, which CadHub is.

The docs state

The Crawler is an automated web scraping program. When given a set of start URLs, it visits and extracts content from those pages. It then visits URLs these pages link to, and the process repeats itself for all linked pages. With little configuration the Crawler can populate and maintain Algolia indices for you by periodically extracting content from your web pages.

You would need to create an Algolia account (something you need to do regardless of how we go about setting this up) and then link it to your Netlify account by following these guidelines.

In my cloned CadHub repo I got search running on the homepage and on a separate search page.

The separate search page is ugly (un-styled) and mainly just a proof of concept... Right now when I search there are no results cataloged in the Algolia indices so no search results are returned, which is where the Crawler would come in.

Obviously I do not have access to the Netlify account to link the Algolia Crawler to (I tried using my own Netlify account but it would not allow me to create the Crawler - kinda makes sense though why it did not work. Only apps hosted on the Netlify account should be authorized for an Algolia implemented "scraping"). Would you be willing to give it a try setting up the Crawler? My thought was just to point the crawler at https://cadhub.xyz/projects and see what it pulls into the indices... I had never heard of it until today, so I am not sure how well it works but on paper it looks really promising.

Cheers!

I [pinged](https://community.redwoodjs.com/t/question-about-cells-in-regards-to-algolia-search-functionality/2755/2) the Redwood forum regarding this project and got some very helpful feedback from Rob and David T. While sorting out how to write a script to populate the Algolia indices I came across [Algolia Crawler](https://www.algolia.com/doc/tools/crawler/getting-started/overview/). It works with the free plan with sites hosted by Netlify, which CadHub is. The docs state > The Crawler is an automated web scraping program. When given a set of start URLs, it visits and extracts content from those pages. It then visits URLs these pages link to, and the process repeats itself for all linked pages. With little configuration the Crawler can populate and maintain Algolia indices for you by periodically extracting content from your web pages. You would need to create an Algolia account (something you need to do regardless of how we go about setting this up) and then link it to your Netlify account by following [these](https://www.algolia.com/doc/tools/crawler/netlify-plugin/quick-start/) guidelines. In my cloned CadHub repo I got search running on the [homepage](https://github.com/Isaac-Tait/cadhub/blob/main/app/web/src/layouts/MainLayout/MainLayout.js) and on a separate [search page](https://github.com/Isaac-Tait/cadhub/tree/main/app/web/src/pages/SearchPage). ![homePage](https://user-images.githubusercontent.com/14932877/153634709-46f218de-6fbc-4873-a8df-82b38c5ca729.png) --- ![SuperUgly](https://user-images.githubusercontent.com/14932877/153634725-f0e14897-3e75-4d72-8bf8-71d3a2299279.png) The separate search page is ugly (un-styled) and mainly just a proof of concept... Right now when I search there are no results cataloged in the Algolia indices so no search results are returned, which is where the Crawler would come in. Obviously I do not have access to the Netlify account to link the Algolia Crawler to (I tried using my own Netlify account but it would not allow me to create the Crawler - kinda makes sense though why it did not work. Only apps hosted on the Netlify account should be authorized for an Algolia implemented "scraping"). Would you be willing to give it a try setting up the Crawler? My thought was just to point the crawler at https://cadhub.xyz/projects and see what it pulls into the indices... I had never heard of it until today, so I am not sure how well it works but on paper it looks really promising. Cheers!

Irev-Dev commented

2022-02-20 07:12:34 +01:00

(Migrated from github.com)

Sorry I've been a little MIA of late, thanks for looking into all this.

I'm not so sure about the crawler though I could probably set it up as an experiment. My concerns are

The netlify hosting is likely to change, as I'd like it to change to deploying with the serverless framework on AWS. I couldn't find anywhere that said it has to be netlify with the free teir?
I'm not sure it's fit for purpose, it does sound easy to set up, but looking at their FAQ

Our data accessible from it original source, or am I miss-interpreting that?

Sorry I've been a little MIA of late, thanks for looking into all this. I'm not so sure about the crawler though I could probably set it up as an experiment. My concerns are - The netlify hosting is likely to change, as I'd like it to change to deploying with the serverless framework on AWS. I couldn't find anywhere that said it has to be netlify with the free teir? - I'm not sure it's fit for purpose, it does sound easy to set up, but looking at their FAQ ![image](https://user-images.githubusercontent.com/29681384/154830959-a5ab8881-1bee-4123-9642-b76a9e72bca4.png) Our data accessible from it original source, or am I miss-interpreting that?

Isaac-Tait commented

2022-02-26 16:24:34 +01:00

(Migrated from github.com)

The netlify hosting is likely to change, as I'd like it to change to deploying with the serverless framework on AWS.

If you are not going to host on Netlify in the future I would prefer to figure out a method that would allow search to work no matter which service you choose to host with.

I am curious though if it does work on the free tier. I might just try and set it up on another project and see...

data accessible from its original source

I did read that too and thought it was kind of odd. I wonder what use case would mean your data was not available from the original source? 🤷🏻‍♂️

> The netlify hosting is likely to change, as I'd like it to change to deploying with the serverless framework on AWS. If you are not going to host on Netlify in the future I would prefer to figure out a method that would allow search to work no matter which service you choose to host with. I am curious though if it does work on the free tier. I might just try and set it up on another project and see... > data accessible from its original source I did read that too and thought it was kind of odd. I wonder what use case would mean your data was not available from the original source? 🤷🏻‍♂️

hrgdavor commented

2022-02-26 17:26:17 +01:00

(Migrated from github.com)

#RANT

All I am hearing here is overengineered cloud crap fluff.
I am used to maintain my own infrastructure and would just make a lucene index for search and be done with it. (update specific doc on project save).

it is all fun and games starting a project with all those fancy services, until you realize you have a huge block of concrete on your legs and can't swim.

You should have at least part of infrastructure that you controll to avoid cloud proveders sucking u dry.
good servers are not that expensive https://www.hetzner.com/dedicated-rootserver

#RANT All I am hearing here is overengineered cloud crap fluff. I am used to maintain my own infrastructure and would just make a lucene index for search and be done with it. (update specific doc on project save). it is all fun and games starting a project with all those fancy services, until you realize you have a huge block of concrete on your legs and can't swim. You should have at least part of infrastructure that you controll to avoid cloud proveders sucking u dry. good servers are not that expensive https://www.hetzner.com/dedicated-rootserver

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: h3n3/cadhub#104