+++ /dev/null
- Apache License\r
- Version 2.0, January 2004\r
- http://www.apache.org/licenses/\r
-\r
- TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\r
-\r
- 1. Definitions.\r
-\r
- "License" shall mean the terms and conditions for use, reproduction,\r
- and distribution as defined by Sections 1 through 9 of this document.\r
-\r
- "Licensor" shall mean the copyright owner or entity authorized by\r
- the copyright owner that is granting the License.\r
-\r
- "Legal Entity" shall mean the union of the acting entity and all\r
- other entities that control, are controlled by, or are under common\r
- control with that entity. For the purposes of this definition,\r
- "control" means (i) the power, direct or indirect, to cause the\r
- direction or management of such entity, whether by contract or\r
- otherwise, or (ii) ownership of fifty percent (50%) or more of the\r
- outstanding shares, or (iii) beneficial ownership of such entity.\r
-\r
- "You" (or "Your") shall mean an individual or Legal Entity\r
- exercising permissions granted by this License.\r
-\r
- "Source" form shall mean the preferred form for making modifications,\r
- including but not limited to software source code, documentation\r
- source, and configuration files.\r
-\r
- "Object" form shall mean any form resulting from mechanical\r
- transformation or translation of a Source form, including but\r
- not limited to compiled object code, generated documentation,\r
- and conversions to other media types.\r
-\r
- "Work" shall mean the work of authorship, whether in Source or\r
- Object form, made available under the License, as indicated by a\r
- copyright notice that is included in or attached to the work\r
- (an example is provided in the Appendix below).\r
-\r
- "Derivative Works" shall mean any work, whether in Source or Object\r
- form, that is based on (or derived from) the Work and for which the\r
- editorial revisions, annotations, elaborations, or other modifications\r
- represent, as a whole, an original work of authorship. For the purposes\r
- of this License, Derivative Works shall not include works that remain\r
- separable from, or merely link (or bind by name) to the interfaces of,\r
- the Work and Derivative Works thereof.\r
-\r
- "Contribution" shall mean any work of authorship, including\r
- the original version of the Work and any modifications or additions\r
- to that Work or Derivative Works thereof, that is intentionally\r
- submitted to Licensor for inclusion in the Work by the copyright owner\r
- or by an individual or Legal Entity authorized to submit on behalf of\r
- the copyright owner. For the purposes of this definition, "submitted"\r
- means any form of electronic, verbal, or written communication sent\r
- to the Licensor or its representatives, including but not limited to\r
- communication on electronic mailing lists, source code control systems,\r
- and issue tracking systems that are managed by, or on behalf of, the\r
- Licensor for the purpose of discussing and improving the Work, but\r
- excluding communication that is conspicuously marked or otherwise\r
- designated in writing by the copyright owner as "Not a Contribution."\r
-\r
- "Contributor" shall mean Licensor and any individual or Legal Entity\r
- on behalf of whom a Contribution has been received by Licensor and\r
- subsequently incorporated within the Work.\r
-\r
- 2. Grant of Copyright License. Subject to the terms and conditions of\r
- this License, each Contributor hereby grants to You a perpetual,\r
- worldwide, non-exclusive, no-charge, royalty-free, irrevocable\r
- copyright license to reproduce, prepare Derivative Works of,\r
- publicly display, publicly perform, sublicense, and distribute the\r
- Work and such Derivative Works in Source or Object form.\r
-\r
- 3. Grant of Patent License. Subject to the terms and conditions of\r
- this License, each Contributor hereby grants to You a perpetual,\r
- worldwide, non-exclusive, no-charge, royalty-free, irrevocable\r
- (except as stated in this section) patent license to make, have made,\r
- use, offer to sell, sell, import, and otherwise transfer the Work,\r
- where such license applies only to those patent claims licensable\r
- by such Contributor that are necessarily infringed by their\r
- Contribution(s) alone or by combination of their Contribution(s)\r
- with the Work to which such Contribution(s) was submitted. If You\r
- institute patent litigation against any entity (including a\r
- cross-claim or counterclaim in a lawsuit) alleging that the Work\r
- or a Contribution incorporated within the Work constitutes direct\r
- or contributory patent infringement, then any patent licenses\r
- granted to You under this License for that Work shall terminate\r
- as of the date such litigation is filed.\r
-\r
- 4. Redistribution. You may reproduce and distribute copies of the\r
- Work or Derivative Works thereof in any medium, with or without\r
- modifications, and in Source or Object form, provided that You\r
- meet the following conditions:\r
-\r
- (a) You must give any other recipients of the Work or\r
- Derivative Works a copy of this License; and\r
-\r
- (b) You must cause any modified files to carry prominent notices\r
- stating that You changed the files; and\r
-\r
- (c) You must retain, in the Source form of any Derivative Works\r
- that You distribute, all copyright, patent, trademark, and\r
- attribution notices from the Source form of the Work,\r
- excluding those notices that do not pertain to any part of\r
- the Derivative Works; and\r
-\r
- (d) If the Work includes a "NOTICE" text file as part of its\r
- distribution, then any Derivative Works that You distribute must\r
- include a readable copy of the attribution notices contained\r
- within such NOTICE file, excluding those notices that do not\r
- pertain to any part of the Derivative Works, in at least one\r
- of the following places: within a NOTICE text file distributed\r
- as part of the Derivative Works; within the Source form or\r
- documentation, if provided along with the Derivative Works; or,\r
- within a display generated by the Derivative Works, if and\r
- wherever such third-party notices normally appear. The contents\r
- of the NOTICE file are for informational purposes only and\r
- do not modify the License. You may add Your own attribution\r
- notices within Derivative Works that You distribute, alongside\r
- or as an addendum to the NOTICE text from the Work, provided\r
- that such additional attribution notices cannot be construed\r
- as modifying the License.\r
-\r
- You may add Your own copyright statement to Your modifications and\r
- may provide additional or different license terms and conditions\r
- for use, reproduction, or distribution of Your modifications, or\r
- for any such Derivative Works as a whole, provided Your use,\r
- reproduction, and distribution of the Work otherwise complies with\r
- the conditions stated in this License.\r
-\r
- 5. Submission of Contributions. Unless You explicitly state otherwise,\r
- any Contribution intentionally submitted for inclusion in the Work\r
- by You to the Licensor shall be under the terms and conditions of\r
- this License, without any additional terms or conditions.\r
- Notwithstanding the above, nothing herein shall supersede or modify\r
- the terms of any separate license agreement you may have executed\r
- with Licensor regarding such Contributions.\r
-\r
- 6. Trademarks. This License does not grant permission to use the trade\r
- names, trademarks, service marks, or product names of the Licensor,\r
- except as required for reasonable and customary use in describing the\r
- origin of the Work and reproducing the content of the NOTICE file.\r
-\r
- 7. Disclaimer of Warranty. Unless required by applicable law or\r
- agreed to in writing, Licensor provides the Work (and each\r
- Contributor provides its Contributions) on an "AS IS" BASIS,\r
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\r
- implied, including, without limitation, any warranties or conditions\r
- of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\r
- PARTICULAR PURPOSE. You are solely responsible for determining the\r
- appropriateness of using or redistributing the Work and assume any\r
- risks associated with Your exercise of permissions under this License.\r
-\r
- 8. Limitation of Liability. In no event and under no legal theory,\r
- whether in tort (including negligence), contract, or otherwise,\r
- unless required by applicable law (such as deliberate and grossly\r
- negligent acts) or agreed to in writing, shall any Contributor be\r
- liable to You for damages, including any direct, indirect, special,\r
- incidental, or consequential damages of any character arising as a\r
- result of this License or out of the use or inability to use the\r
- Work (including but not limited to damages for loss of goodwill,\r
- work stoppage, computer failure or malfunction, or any and all\r
- other commercial damages or losses), even if such Contributor\r
- has been advised of the possibility of such damages.\r
-\r
- 9. Accepting Warranty or Additional Liability. While redistributing\r
- the Work or Derivative Works thereof, You may choose to offer,\r
- and charge a fee for, acceptance of support, warranty, indemnity,\r
- or other liability obligations and/or rights consistent with this\r
- License. However, in accepting such obligations, You may act only\r
- on Your own behalf and on Your sole responsibility, not on behalf\r
- of any other Contributor, and only if You agree to indemnify,\r
- defend, and hold each Contributor harmless for any liability\r
- incurred by, or claims asserted against, such Contributor by reason\r
- of your accepting any such warranty or additional liability.\r
-\r
- END OF TERMS AND CONDITIONS\r
-\r
- APPENDIX: How to apply the Apache License to your work.\r
-\r
- To apply the Apache License to your work, attach the following\r
- boilerplate notice, with the fields enclosed by brackets "[]"\r
- replaced with your own identifying information. (Don't include\r
- the brackets!) The text should be enclosed in the appropriate\r
- comment syntax for the file format. We also recommend that a\r
- file or class name and description of purpose be included on the\r
- same "printed page" as the copyright notice for easier\r
- identification within third-party archives.\r
-\r
- Copyright 2021 Developer's House\r
-\r
- Licensed under the Apache License, Version 2.0 (the "License");\r
- you may not use this file except in compliance with the License.\r
- You may obtain a copy of the License at\r
-\r
- http://www.apache.org/licenses/LICENSE-2.0\r
-\r
- Unless required by applicable law or agreed to in writing, software\r
- distributed under the License is distributed on an "AS IS" BASIS,\r
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\r
- See the License for the specific language governing permissions and\r
- limitations under the License.\r
+++ /dev/null
-# Todo\r
-\r
-## GitHub\r
-\r
-- PR and Issue template\r
-- Contributing GUIDE\r
-\r
-### Readme\r
-\r
-- Features\r
-- Explaination of each components\r
-- How to use (dev, prod)\r
-- Contributing\r
-- Sponsor?
\ No newline at end of file
--- /dev/null
+syntax = "proto3";
+
+import "common/management/nova.management.v1alpha.proto";
+package nova.management.rpc.v1alpha;
\ No newline at end of file
+++ /dev/null
-#Β Nova documentation
\ No newline at end of file
+++ /dev/null
-# Common packages
-
-Nova exposes common packages for all the different projects in the mono-repo.
-Since Nova uses two different languages, it has two different common libraries,
-one built in rust in [the common/rust](../common/rust) directory and one in Go
-in [the common/](../common) directory.
\ No newline at end of file
+++ /dev/null
-# Webhook
-
-> TD;TR The webhook component is an implementation of the discord interaction webhooks; You can either use the Gateway or the Webhooks. The webhooks __require__ an external https endpoint to work.
-
-The webhook source code is located in the [webhook](../../webhook) folder and is implemented in Rust. It's a simple http web server which implements the webhook signature verification and deserialization. Like the gateway, the messages are redirected using the [relaying system](../common#relaying_trait).
-
-The signature verification is done using libsodium via the libsodium-sys trait.
-Subsequently, it uses code marked as "unsafe" in rust. It's built into the binary statically. Any route can be used to receive webhook messages.
\ No newline at end of file
--- /dev/null
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+Copyright 2021 Developer's House
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
--- /dev/null
+{
+ "label": "Deployment",
+ "position": 3
+}
\ No newline at end of file
--- /dev/null
+# Production guide
\ No newline at end of file
--- /dev/null
+{
+ "label": "Internals",
+ "position": 99999
+}
\ No newline at end of file
--- /dev/null
+{
+ "label": "Components",
+ "position": 4
+}
\ No newline at end of file
--- /dev/null
+---
+title: Gateway
+position: 1
+---
+
+The gateway if the component responsible for the real-time events
+using the Discord Gateway API (websocket), the component is written
+in rust using [`tokio-tungstenite`](https://github.com/snapview/tokio-tungstenite) and his job is to simply maintain
+a gateway connection (heartbeat, reconnect, etc...) and send events
+directly to the cache using nats.
+
+The project is located in the `gateway` folder at the root
+of the GitHub repo and is published using a statically linked
+library (except libc) and docker images.
+
+## Structure
+
+Internally, the gateway is divided in multiple rust modules
+
+### Connection (src/connection)
+
+This module implements a [`futures_core::stream::Stream` trait](https://docs.rs/futures-core/0.3.17/futures_core/stream/trait.Stream.html)
+and [`futures::sink::Sink` trait](https://docs.rs/futures/0.3.17/futures/sink/trait.Sink.html) that streams deserialized discord
+packets to the stream. It does not implement any reconnect mechanism and returns
+an error when the connection encounters a deserialization error or i/o error.
+If the connection is closed, the stream simply closes.
+
+You can send any `BaseMessage` struct through the connection and receive
+any `BaseMessage` struct.
+
+
+### Payloads (src/payload)
+
+This module implements the deserialization of discord packets
--- /dev/null
+---
+sidebar_position: 1
+---
+
+# Nova Internals
+
+## Definitions
+
+### Cache
+
+A cache is an instance of the `cache` program, it managed everything that is related to data-management
+of the discord events, webhooks and more.
+
+### Gateway
+*todo*
+### Webhook
+*todo*
+###
+
+## Communication
+
+```
+ βββββββββββββββββ βββββββββββββββββββββ
+ β Gateway β event(s) β β βββββββββββββββββββββββββ
+ββββΊβ (gateway) βββββββββββββββΊβ ββββββββββββββββββββΊβ β
+β βββββββββββββββββ (nats queue) β Cache instance β (redis channel) β Caching server β
+β β (cache) β β (redis) β
+β βββββββββββββββββ event(s) β βββββββββββββββββββββ€ β
+β β Webhook βββββββββββββββΊβ β βββββββββββββββββββββββββ
+β β (webhook) β (nats queue) βββββββββββββββββββ¬ββ
+β βββββββββββββββββ β² β
+β cache requests β β event(s)
+β (grpc) β β (nats queue)
+β β βΌ
+β βββββ΄ββββββββββββββββ
+ββββββββββββββββββββββββββββββββββββ€ User programs β
+ gateway commands β (client libs) β
+ (grpc) βββββββββββββββββββββ
+```
+
+> We try to use protocol buffers everywhere even inside the event broker (nats),
+> this ensures we have the same date representation across all the programs written
+> in different languages
+
+> The only dependency of the nova architecture is a reliable nats & redis cluster for now
+
+#### Protocols
+
+All the protocols a detailed in the [protocols folder](protocols)
+
+####
\ No newline at end of file
--- /dev/null
+# Protocols
+
+This section of the internal documentation describes the protocols used by nova
+to communicate between all the components.
+
+
+*
\ No newline at end of file
--- /dev/null
+# Introduction
+
+## What is nova ?
+
+Nova is a complete framework for building reliable and stable discord bots
+using a services based system to operate all the components needed to operate
+a discord such as the discord gateway, new discord webhooks for receiving interactions.
+Using a traditional infrastructure (sharder / process), we can quickly reach bottlenecks
+when we scale to multiple thousands of shards. Nova is a way to avoid these problems by
+building a scale-first discord framework that creates an environment that allow
+better testing, reliability and operations easier.
+
+### Advantages
+
+With the help of Nova, you can achieve a number of things, such as
+
+* Scaling of workers independent of the number of shards
+* zero-login updates
+* Automatic shards scaling
+* Shared cache for the whole bot (without broadcastEval and other unsafe methods)
+* Stateless workers, easier to test
+* Distributed rest rate-limiting
+* Easier fine-tuned monitoring using cloud-native technologies
+* Languages agnostic apis
+
+### How did we solve this ?
+
+Nova separates the gateway into multiple smaller components corresponding to multiple
+discord apis
--- /dev/null
+---
+sidebar_position: 2
+---
+
+# 5 Minutes quickstart
+
+This page shows you how to start a new nova project
+under five minutes using a typescript project,
+hold tight this is going to be fast.
+
+## Requirements
+
+* A discord bot application available
+* [Docker](https://docker.io/) (or alternatives) available to you.
+* A domain name / [ngrok.io](https://ngrok.com/) domain (for webhooks only)
+
+> If you are deploying nova to production, consider following the
+> production guide instead.
+
+## Setting up a nova instance
+
+++ /dev/null
-# Nova Internals - Management
-
-In order to run in multiple environments, Nova uses a scheduling system to balance shards between "cluster" instances.
-Each cluster instance is in fact a gateway instance. All the management is handled in the "manager" component.
-The discord shards are dynamically scaled and restarted if needed.
\ No newline at end of file
+++ /dev/null
-# Workspace/System structure
-
-##Β System structure
-
-In the nova repository, two different types of projects exist,
-
-* The management projects primarly in Go \
- They manage the other components of the nova infrastructure.
-* The data-path projects \
- They handle all the data transfer / management.
-
-###Β Gateway
-
-> The gateway interfaces with the discord gateway to retrive events in real time
-It's implemented in rust and is in the gateway folder.
\ No newline at end of file
+++ /dev/null
-load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
-
-go_library(
- name = "manager_lib",
- srcs = ["main.go"],
- importpath = "github.com/discordnova/nova/manager",
- visibility = ["//visibility:private"],
- deps = [
- "//manager/lib",
- "//manager/lib/scheduler",
- "@com_github_buraksezer_consistent//:consistent",
- ],
-)
-
-go_binary(
- name = "manager",
- embed = [":manager_lib"],
- visibility = ["//visibility:public"],
-)
+++ /dev/null
-load("@io_bazel_rules_go//go:def.bzl", "go_library")
-
-go_library(
- name = "lib",
- srcs = [
- "config.go",
- "main.go",
- ],
- importpath = "github.com/discordnova/nova/manager/lib",
- visibility = ["//visibility:public"],
- deps = [
- "//common/management",
- "//manager/lib/services",
- "@org_golang_google_grpc//:go_default_library",
- ],
-)
+++ /dev/null
-package lib
-
-
-type GrpcConfig struct {
- Server string
-}
-
-type Config struct {
- Grpc GrpcConfig
-}
+++ /dev/null
-package lib
-
-import (
- "github.com/discordnova/nova/common/management"
- "github.com/discordnova/nova/manager/lib/services"
- "google.golang.org/grpc"
- "log"
- "net"
-)
-
-func StartGrpcServices(config *Config) {
- lis, err := net.Listen("tcp", config.Grpc.Server)
-
- if err != nil {
- log.Fatalf("failed to start the grpc services: %v", err)
- }
- var opts []grpc.ServerOption
-
- grpcServer := grpc.NewServer(opts...)
-
- management.RegisterManagementServiceServer(grpcServer, services.CreateStatusServerService())
-
- err = grpcServer.Serve(lis)
- if err != nil {
- log.Fatalf("failed to start the grpc server: %v", err)
- }
-}
+++ /dev/null
-load("@io_bazel_rules_go//go:def.bzl", "go_library")
-
-go_library(
- name = "scheduler",
- srcs = ["main.go"],
- importpath = "github.com/discordnova/nova/manager/lib/scheduler",
- visibility = ["//visibility:public"],
- deps = [
- "@com_github_buraksezer_consistent//:consistent",
- "@com_github_cespare_xxhash//:xxhash",
- ],
-)
+++ /dev/null
-package scheduler
-
-import (
- "github.com/buraksezer/consistent"
- "github.com/cespare/xxhash"
- "log"
-)
-
-// consistent package doesn't provide a default hashing function.
-// You should provide a proper one to distribute keys/members uniformly.
-type hasher struct{}
-
-func (h hasher) Sum64(data []byte) uint64 {
- // you should use a proper hash function for uniformity.
- return xxhash.Sum64(data)
-}
-
-type State struct {
- Members []consistent.Member
- CurrentSize int
- MaxConcurrency int
-}
-
-type Scheduler struct {
- state State
- hashRing *consistent.Consistent
-}
-
-func CreateScheduler(state State) *Scheduler {
- config := consistent.Config{
- Hasher: hasher{},
- PartitionCount: 271,
- ReplicationFactor: 20,
- Load: 1.25,
- }
-
- return &Scheduler{
- state: state,
- hashRing: consistent.New(state.Members, config),
- }
-}
-
-// Scale changes the size of Discord shards
-func (s *Scheduler) Scale(newScale int) {
- log.Printf("scaling from %d shards to %d shards", s.state.CurrentSize, newScale)
- // todo: take the scale mutex
-
- // we should handle all the sessions restart
- // and the re-scheduling of the shards including rate-limiting
- // first, we remove all the shards instances from the clusters
- for i := 0; i < s.state.CurrentSize; i++ {
- // todo: call the rpc for stopping the shard on the given cluster
- cluster := s.hashRing.LocateKey([]byte{byte(i)})
- log.Printf("sending stop shard to shard %d on cluster %s", i, cluster)
- }
- s.state.CurrentSize = newScale
-
-
- // after we stopped the shard, we re-start it with the new number of shards
- // since all the shard must be restarted after a scale operation
- // but before that, we need to allocate all the shards
- for i := 0; i < s.state.CurrentSize; i++ {
- cluster := s.hashRing.LocateKey([]byte{byte(i)})
- // todo: call the rpc for allocate the shard on the given cluster
- log.Printf("sending allocate shard to shard %d on %s with [%d, %d] as sharding key", i, cluster, i, newScale)
- }
-
- // we start the first shard, this represents the "pilot" shard which gives us the
- // required information, such as the max_concurrency.
- // todo: call the rpc to start the first shard
-
- // according to the max_concurrency system, we must
- // start the shards sequentially.
- for i := 0; i < s.state.MaxConcurrency; i++ {
- log.Printf("starting bucket %d", i)
- for j := 0; j < newScale; j++ {
- // if the shard corresponds to the current starting bucket
- if j % s.state.MaxConcurrency == i {
- cluster := s.hashRing.LocateKey([]byte{byte(j)})
- // todo: call the start rpc on the given cluster
- log.Printf("%d: sending the start command to the shard %d on the %s cluster", i, j, cluster)
- }
- }
- }
-
- // todo: release the scale mutex
-}
-
-// RemoveNode removes a cluster from the consistent hash ring and relocate the shards
-func (s *Scheduler) RemoveNode() {
-
-}
-
-// AddNode adds a node to the consistent hash ring and relocate the shards
-func (s *Scheduler) AddNode() {}
-
-// GetShardLocation returns the supposed name of the cluster containing a Shard
-func (s *Scheduler) GetShardLocation() string {
- return ""
-}
+++ /dev/null
-load("@io_bazel_rules_go//go:def.bzl", "go_library")
-
-go_library(
- name = "services",
- srcs = ["status-server.go"],
- importpath = "github.com/discordnova/nova/manager/lib/services",
- visibility = ["//visibility:public"],
- deps = ["//common/management"],
-)
+++ /dev/null
-package services
-
-import (
- "context"
- "fmt"
- "github.com/discordnova/nova/common/management"
-)
-
-type statusServer struct {
- management.UnimplementedManagementServiceServer
-}
-
-func CreateStatusServerService() management.ManagementServiceServer {
- return &statusServer{}
-}
-
-func (s *statusServer) GetGlobalClusterStatus(context context.Context, _ *management.Empty) (*management.GlobalClusterStatusResponse, error) {
- return nil, fmt.Errorf("not implemented")
-}
-func (s *statusServer) GetClusterStatus(context context.Context, request *management.ClusterStatusRequest) (*management.ClusterStatusResponse, error) {
- return nil, fmt.Errorf("not implemented")
-}
-
-func (s *statusServer) GetShardStatus(context context.Context, request *management.ShardStatusRequest) (*management.ShardStatusResponse, error) {
- return nil, fmt.Errorf("not implemented")
-}
+++ /dev/null
-package main
-
-import (
- "github.com/buraksezer/consistent"
- "github.com/discordnova/nova/manager/lib"
- "github.com/discordnova/nova/manager/lib/scheduler"
- "log"
-)
-
-type myMember string
-
-func (m myMember) String() string {
- return string(m)
-}
-
-func main () {
- scheduler.CreateScheduler(scheduler.State{
- Members: []consistent.Member{myMember("cluster1"), myMember("cluster2"), myMember("cluster3"), myMember("cluster4")},
- CurrentSize: 2,
- MaxConcurrency: 5,
- }).Scale(10)
- log.Printf("starting grpc")
- lib.StartGrpcServices(&lib.Config{
- Grpc: lib.GrpcConfig{Server: "0.0.0.0:8053"},
- })
-}
\ No newline at end of file
+++ /dev/null
-// How does this works ?\r
-// Every request, the proxy (envoy) requests the rate-limiting service if\r
-// the requested route bucket or global rate-limit is hit.\r
-\r
-syntax = "proto3";\r
-package nova.ratelimit.v1;\r
-\r
-// The reponse of a RatelimitRequest, it includes the status of the reponse and\r
-// the bucket informations.\r
-message RatelimitResponse {\r
- enum Status {\r
- OK = 0;\r
- RATELIMITED = 1;\r
- GLOBAL_RATELIMITED = 2;\r
- }\r
- Status status = 1;\r
- bool updateAsked = 2;\r
-}\r
-\r
-// Requests the ratelimit status of a route request, it also takes the \r
-// indentifiables of the request in question.\r
-message RatelimitRequest {\r
- string routeName = 1;\r
- repeated string indentifiables = 2;\r
-}\r
-\r
-// Used when "updateAsked" is sed to true\r
-// this means the bucket is unknown to the ratelimit server.\r
-message CreateBucketData {\r
- RatelimitRequest request = 1;\r
- int32 limit = 2;\r
- int32 remaining = 3;\r
- int32 reset = 4;\r
-}\r
-\r
-service RatelimitService {\r
- rpc GetRatelimitStatus (RatelimitRequest) returns (RatelimitResponse);\r
- rpc CreateBucket (CreateBucketData) returns (CreateBucketData);\r
-}
\ No newline at end of file
--- /dev/null
+// How does this works ?
+// Every request, the proxy (envoy) requests the rate-limiting service if
+// the requested route bucket or global rate-limit is hit.
+
+syntax = "proto3";
+package nova.ratelimit.v1;
+
+// The reponse of a RatelimitRequest, it includes the status of the reponse and
+// the bucket informations.
+message RatelimitResponse {
+ enum Status {
+ OK = 0;
+ RATELIMITED = 1;
+ GLOBAL_RATELIMITED = 2;
+ }
+ Status status = 1;
+ bool updateAsked = 2;
+}
+
+// Requests the ratelimit status of a route request, it also takes the
+// indentifiables of the request in question.
+message RatelimitRequest {
+ string routeName = 1;
+ repeated string indentifiables = 2;
+}
+
+// Used when "updateAsked" is sed to true
+// this means the bucket is unknown to the ratelimit server.
+message CreateBucketData {
+ RatelimitRequest request = 1;
+ int32 limit = 2;
+ int32 remaining = 3;
+ int32 reset = 4;
+}
+
+service RatelimitService {
+ rpc GetRatelimitStatus (RatelimitRequest) returns (RatelimitResponse);
+ rpc CreateBucket (CreateBucketData) returns (CreateBucketData);
+}
\ No newline at end of file
+++ /dev/null
-monitoring:
- enabled: true
- address: "0.0.0.0"
- port: 5000
-nats:
- host: "nova-infra-nats"
-webhook:
- server:
- port: 8080
- address: "0.0.0.0"
- discord:
- client_id: 0
- public_key: ""
\ No newline at end of file
projects / matthieu / nova.git / commitdiff