Skip to content

Commit 1a69f37

Browse files
maltesanderrazvan
maltesander
and
razvan
committed
Stabilize hdfs cluster install (#158)
## Description - adapted scripts for formatting namenodes - code clean up based on PR #150 closes #153 #147 Co-authored-by: Malte Sander <malte.sander.it@gmail.com> Co-authored-by: Razvan-Daniel Mihai <84674+razvan@users.noreply.github.com>
1 parent 3cee26c commit 1a69f37

File tree

17 files changed

+650
-412
lines changed

17 files changed

+650
-412
lines changed

‎CHANGELOG.md

+4-1
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ All notable changes to this project will be documented in this file.
1010
- Reconciliation errors are now reported as Kubernetes events ([#130]).
1111
- Use cli argument `watch-namespace` / env var `WATCH_NAMESPACE` to specify
1212
a single namespace to watch ([#134]).
13+
- Config builder for `hdfs-site.xml` and `core-site.xml` ([#150]).
14+
- Discovery configmap that exposes the namenode services for clients to connect ([#150]).
15+
- Documented service discovery for namenodes ([#150]).
1316

1417
### Changed
1518

@@ -20,6 +23,7 @@ All notable changes to this project will be documented in this file.
2023
[#130]: https://github.com/stackabletech/hdfs-operator/pull/130
2124
[#134]: https://github.com/stackabletech/hdfs-operator/pull/134
2225
[#148]: https://github.com/stackabletech/hdfs-operator/pull/148
26+
[#150]: https://github.com/stackabletech/hdfs-operator/pull/150
2327

2428
## [0.3.0] - 2022-02-14
2529

@@ -37,7 +41,6 @@ All notable changes to this project will be documented in this file.
3741

3842
## [0.2.0] - 2021-11-12
3943

40-
4144
- `operator-rs` `0.3.0` → `0.4.0` ([#20]).
4245
- Adapted pod image and container command to docker image ([#20]).
4346
- Adapted documentation to represent new workflow with docker images ([#20]).

‎Cargo.lock

-1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎deploy/config-spec/properties.yaml

+7-28
Original file line numberDiff line numberDiff line change
@@ -73,14 +73,11 @@ properties:
7373
datatype:
7474
type: "string"
7575
unit: *unitUri
76-
defaultValues:
77-
- fromVersion: "0.0.0"
78-
value: "0.0.0.0:9000"
7976
roles:
8077
- name: "namenode"
81-
required: true
78+
required: false
8279
- name: "datanode"
83-
required: true
80+
required: false
8481
asOfVersion: "0.0.0"
8582
description: "The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem."
8683

@@ -92,9 +89,6 @@ properties:
9289
file: "hdfs-site.xml"
9390
datatype:
9491
type: "string"
95-
defaultValues:
96-
- fromVersion: "3.0.0"
97-
value: "0.0.0.0:9870"
9892
roles:
9993
- name: "namenode"
10094
required: false
@@ -109,12 +103,9 @@ properties:
109103
file: "hdfs-site.xml"
110104
datatype:
111105
type: "string"
112-
defaultValues:
113-
- fromVersion: "3.0.0"
114-
value: "0.0.0.0:9867"
115106
roles:
116107
- name: "datanode"
117-
required: true
108+
required: false
118109
asOfVersion: "0.0.0"
119110
description: "The datanode ipc server address and port."
120111

@@ -126,12 +117,9 @@ properties:
126117
file: "hdfs-site.xml"
127118
datatype:
128119
type: "string"
129-
defaultValues:
130-
- fromVersion: "3.0.0"
131-
value: "0.0.0.0:9866"
132120
roles:
133121
- name: "datanode"
134-
required: true
122+
required: false
135123
asOfVersion: "0.0.0"
136124
description: "The datanode server address and port for data transfer."
137125

@@ -143,12 +131,9 @@ properties:
143131
file: "hdfs-site.xml"
144132
datatype:
145133
type: "string"
146-
defaultValues:
147-
- fromVersion: "3.0.0"
148-
value: "0.0.0.0:9864"
149134
roles:
150135
- name: "datanode"
151-
required: true
136+
required: false
152137
asOfVersion: "0.0.0"
153138
description: "The datanode http server address and port."
154139

@@ -160,11 +145,8 @@ properties:
160145
file: "hdfs-site.xml"
161146
datatype:
162147
type: "string"
163-
defaultValues:
164-
- fromVersion: "3.0.0"
165-
value: "0.0.0.0:8480"
166148
roles:
167-
- name: "journanode"
149+
- name: "journalnode"
168150
required: false
169151
asOfVersion: "0.0.0"
170152
description: "The address and port the JournalNode HTTP server listens on. If the port is 0 then the server will start on a free port."
@@ -177,11 +159,8 @@ properties:
177159
file: "hdfs-site.xml"
178160
datatype:
179161
type: "string"
180-
defaultValues:
181-
- fromVersion: "3.0.0"
182-
value: "0.0.0.0:8481"
183162
roles:
184-
- name: "journanode"
163+
- name: "journalnode"
185164
required: false
186165
asOfVersion: "0.0.0"
187166
description: "The address and port the JournalNode HTTPS server listens on. If the port is 0 then the server will start on a free port."

‎deploy/helm/hdfs-operator/configs/properties.yaml

+7-28
Original file line numberDiff line numberDiff line change
@@ -73,14 +73,11 @@ properties:
7373
datatype:
7474
type: "string"
7575
unit: *unitUri
76-
defaultValues:
77-
- fromVersion: "0.0.0"
78-
value: "0.0.0.0:9000"
7976
roles:
8077
- name: "namenode"
81-
required: true
78+
required: false
8279
- name: "datanode"
83-
required: true
80+
required: false
8481
asOfVersion: "0.0.0"
8582
description: "The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem."
8683

@@ -92,9 +89,6 @@ properties:
9289
file: "hdfs-site.xml"
9390
datatype:
9491
type: "string"
95-
defaultValues:
96-
- fromVersion: "3.0.0"
97-
value: "0.0.0.0:9870"
9892
roles:
9993
- name: "namenode"
10094
required: false
@@ -109,12 +103,9 @@ properties:
109103
file: "hdfs-site.xml"
110104
datatype:
111105
type: "string"
112-
defaultValues:
113-
- fromVersion: "3.0.0"
114-
value: "0.0.0.0:9867"
115106
roles:
116107
- name: "datanode"
117-
required: true
108+
required: false
118109
asOfVersion: "0.0.0"
119110
description: "The datanode ipc server address and port."
120111

@@ -126,12 +117,9 @@ properties:
126117
file: "hdfs-site.xml"
127118
datatype:
128119
type: "string"
129-
defaultValues:
130-
- fromVersion: "3.0.0"
131-
value: "0.0.0.0:9866"
132120
roles:
133121
- name: "datanode"
134-
required: true
122+
required: false
135123
asOfVersion: "0.0.0"
136124
description: "The datanode server address and port for data transfer."
137125

@@ -143,12 +131,9 @@ properties:
143131
file: "hdfs-site.xml"
144132
datatype:
145133
type: "string"
146-
defaultValues:
147-
- fromVersion: "3.0.0"
148-
value: "0.0.0.0:9864"
149134
roles:
150135
- name: "datanode"
151-
required: true
136+
required: false
152137
asOfVersion: "0.0.0"
153138
description: "The datanode http server address and port."
154139

@@ -160,11 +145,8 @@ properties:
160145
file: "hdfs-site.xml"
161146
datatype:
162147
type: "string"
163-
defaultValues:
164-
- fromVersion: "3.0.0"
165-
value: "0.0.0.0:8480"
166148
roles:
167-
- name: "journanode"
149+
- name: "journalnode"
168150
required: false
169151
asOfVersion: "0.0.0"
170152
description: "The address and port the JournalNode HTTP server listens on. If the port is 0 then the server will start on a free port."
@@ -177,11 +159,8 @@ properties:
177159
file: "hdfs-site.xml"
178160
datatype:
179161
type: "string"
180-
defaultValues:
181-
- fromVersion: "3.0.0"
182-
value: "0.0.0.0:8481"
183162
roles:
184-
- name: "journanode"
163+
- name: "journalnode"
185164
required: false
186165
asOfVersion: "0.0.0"
187166
description: "The address and port the JournalNode HTTPS server listens on. If the port is 0 then the server will start on a free port."

‎deploy/manifests/configmap.yaml

+7-28
Original file line numberDiff line numberDiff line change
@@ -77,14 +77,11 @@ data:
7777
datatype:
7878
type: "string"
7979
unit: *unitUri
80-
defaultValues:
81-
- fromVersion: "0.0.0"
82-
value: "0.0.0.0:9000"
8380
roles:
8481
- name: "namenode"
85-
required: true
82+
required: false
8683
- name: "datanode"
87-
required: true
84+
required: false
8885
asOfVersion: "0.0.0"
8986
description: "The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem."
9087
@@ -96,9 +93,6 @@ data:
9693
file: "hdfs-site.xml"
9794
datatype:
9895
type: "string"
99-
defaultValues:
100-
- fromVersion: "3.0.0"
101-
value: "0.0.0.0:9870"
10296
roles:
10397
- name: "namenode"
10498
required: false
@@ -113,12 +107,9 @@ data:
113107
file: "hdfs-site.xml"
114108
datatype:
115109
type: "string"
116-
defaultValues:
117-
- fromVersion: "3.0.0"
118-
value: "0.0.0.0:9867"
119110
roles:
120111
- name: "datanode"
121-
required: true
112+
required: false
122113
asOfVersion: "0.0.0"
123114
description: "The datanode ipc server address and port."
124115
@@ -130,12 +121,9 @@ data:
130121
file: "hdfs-site.xml"
131122
datatype:
132123
type: "string"
133-
defaultValues:
134-
- fromVersion: "3.0.0"
135-
value: "0.0.0.0:9866"
136124
roles:
137125
- name: "datanode"
138-
required: true
126+
required: false
139127
asOfVersion: "0.0.0"
140128
description: "The datanode server address and port for data transfer."
141129
@@ -147,12 +135,9 @@ data:
147135
file: "hdfs-site.xml"
148136
datatype:
149137
type: "string"
150-
defaultValues:
151-
- fromVersion: "3.0.0"
152-
value: "0.0.0.0:9864"
153138
roles:
154139
- name: "datanode"
155-
required: true
140+
required: false
156141
asOfVersion: "0.0.0"
157142
description: "The datanode http server address and port."
158143
@@ -164,11 +149,8 @@ data:
164149
file: "hdfs-site.xml"
165150
datatype:
166151
type: "string"
167-
defaultValues:
168-
- fromVersion: "3.0.0"
169-
value: "0.0.0.0:8480"
170152
roles:
171-
- name: "journanode"
153+
- name: "journalnode"
172154
required: false
173155
asOfVersion: "0.0.0"
174156
description: "The address and port the JournalNode HTTP server listens on. If the port is 0 then the server will start on a free port."
@@ -181,11 +163,8 @@ data:
181163
file: "hdfs-site.xml"
182164
datatype:
183165
type: "string"
184-
defaultValues:
185-
- fromVersion: "3.0.0"
186-
value: "0.0.0.0:8481"
187166
roles:
188-
- name: "journanode"
167+
- name: "journalnode"
189168
required: false
190169
asOfVersion: "0.0.0"
191170
description: "The address and port the JournalNode HTTPS server listens on. If the port is 0 then the server will start on a free port."

‎docs/modules/ROOT/nav.adoc

+2
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,5 @@
22
* xref:configuration.adoc[]
33
* xref:usage.adoc[]
44
* xref:implementation.adoc[]
5+
* Concepts
6+
** xref:discovery.adoc[]

‎docs/modules/ROOT/pages/discovery.adoc

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
:clusterName: \{clusterName\}
2+
:namespace: \{namespace\}
3+
4+
= Discovery
5+
6+
The Stackable Operator for Apache HDFS publishes a discovery https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#configmap-v1-core[`ConfigMap`], which exposes a client configuration bundle that allows access to the Apache HDFS cluster.
7+
8+
== Example
9+
10+
Given the following HDFS cluster:
11+
12+
[source,yaml,subs="normal,callouts"]
13+
----
14+
apiVersion: hdfs.stackable.tech/v1alpha1
15+
kind: HdfsCluster
16+
metadata:
17+
name: {clusterName} # <1>
18+
namespace: {namespace} # <2>
19+
spec:
20+
namenode:
21+
roleGroups:
22+
default: # <3>
23+
[...]
24+
----
25+
<1> The name of the HDFS cluster, which is also the name of the created discovery `ConfigMap`.
26+
<2> The namespace of the discovery `ConfigMap`.
27+
<3> A role group name of the `namenode` role.
28+
29+
The resulting discovery `ConfigMap` is located at `{namespace}/{clusterName}`.
30+
31+
== Contents
32+
33+
The `ConfigMap` data values are formatted as Hadoop XML files which allows simple mounting of that ConfigMap into pods that require access to HDFS.
34+
35+
`core-site.xml`::
36+
Contains the `fs.DefaultFS` which defaults to `hdfs://{clusterName}/`.
37+
38+
`hdfs-site.xml`::
39+
Contains the `dfs.namenode.*` properties for `rpc` and `http` addresses for the `namenodes` as well as the `dfs.nameservices` property which defaults to `hdfs://{clusterName}/`.

‎docs/modules/ROOT/pages/usage.adoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ xref:home::monitoring.adoc[] for more details.
9191

9292
The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).
9393

94-
IMPORTANT: Overriding certain properties can lead to faulty clusters. In general this means, do not change ports, hostnames or properties related to high-availability or security.
94+
IMPORTANT: Overriding certain properties can lead to faulty clusters. In general this means, do not change ports, hostnames or properties related to data dirs, high-availability or security.
9595

9696
=== Configuration Properties
9797

‎examples/simple-hdfs-cluster.yaml

+3-3
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ metadata:
2929
spec:
3030
version: 3.2.2
3131
zookeeperConfigMapName: simple-hdfs-znode
32-
dfsReplication: 3
32+
dfsReplication: 1
3333
log4j: |-
3434
# Define some default values that can be overridden by system properties
3535
hadoop.root.logger=INFO,console
@@ -57,11 +57,11 @@ spec:
5757
selector:
5858
matchLabels:
5959
kubernetes.io/os: linux
60-
replicas: 3
60+
replicas: 1
6161
journalNodes:
6262
roleGroups:
6363
default:
6464
selector:
6565
matchLabels:
6666
kubernetes.io/os: linux
67-
replicas: 3
67+
replicas: 1

0 commit comments

Comments
 (0)