Beruflich Dokumente
Kultur Dokumente
Orange Labs
Products and Services
(OLPS)
Communication and Services direction (COMSERV)
Solutions for Voice and Quality department (SVQ)
ETSI2012.Allrightsreserved
Summary
Chapter1
h
Chapter2
Chapter3
Chapter4
p
ETSI2012.Allrightsreserved
Subjectiveassessmentofnoisereduction
b
f
d
Objectiveevaluationofnoisereduction
Fundamentalsetuptocollectrecordings
Analysisofmeasurementofmobilephones
y
p
TheBasis:ITU
TheBasis:ITUTRec.P.835
TRec.P.835(11/2003)
Subjectivetestmethodologyforevaluatingspeechcommunicationsystemsthatincludenoisesuppressionalgorithm
Subjectiveevaluationofnoisereducedsignals:abidimensional
issue
Someusersfocusonbackgroundnoise(amountanddistortion)
Someusersfocusonthespeechsignal(distortion)
PrincipleofP.835:todrawlisternersattentiononbothtypesof
distortionbeforeprovidinganoverallqualityscore
Listenerssuccessivelyratetheprocessedsignalon
thespeechsignaldistortion,
thebackgroundnoiseintrusiveness
g
theoverallquality
Sub-sample 1
Signal or background noise rating
Sub-sample 2
Background noise or signal rating
Vote 2
BACKGROUND or SIGNAL
Vote 1
SIGNAL or BACKGROUND
ETSI2012.Allrightsreserved
Sub-sample 3
Overall quality rating
Vote 3
OVERALL QUALITY
Chapter2
Chapter
2
Objectiveevaluationofnoise
reduction
ETSI2012.Allrightsreserved
ETSIEG202396
ETSIEG
2023963
S
SpeechqualityperformanceinthepresenceofbackgroundnoisePart3:Backgroundnoisetransmission
h
lit
f
i th
fb k
d i P t3 B k
d i t
i i
objectivetestmethods
bj ti t t
th d
ObjectivetestmethoddevelopedtocalculateobjectiveMOS
forspeech,noiseandtheoverallqualityofatransmitted
for speech, noise and the overall quality of a transmitted
signalcontainingspeechandbackgroundnoise
similarlytoP.835:estimatedSMOS,NMOSandGMOS
basedonanaurallyadequateanalysis
useofP.501frenchsentencesforWB
use of P 501 English sentences for NB
useofP.501EnglishsentencesforNB
needs3inputsignals:clean,noisy(clean+backgroundnoise),processed
Generallyapplicabletonarrowbandandwidebandterminals
G
ll
li bl t
b d d id b d t
i l
Tobeusedinconjunctionwiththebackgroundnoisesdefined
d
id d i EG 202 396 1
andprovidedinEG2023961
ETSI2012.Allrightsreserved
ETSITS103106(2012
(201208)
S
Speechqualityperformanceinthepresenceofbackgroundnoise:Backgroundnoisetransmissionformobileterminals
h
lit
f
i th
fb k
d i B k
d i t
i i f
bil t
i l objectivetestmethods
bj ti t t
th d
Basedonnewsubjectivetestsconductedwithmodernmobile
terminalsin3GPP
terminals in 3GPP
DifferentreferencesystemcomparedtoEG2023963
Onlymobileterminalsused
useofnewspeechmaterial
ModificationofEG2023963:
NeuralnetworkinsteadofregressionforSMOScalculation
N
l
ki
d f
i f S MOS l l i
NewmappingforS, N andGMOSbasedonnewsubjectivetests
Differences to EG 2023963:
DifferencestoEG2023963:
restrictedscopecomparedtoEG2023963
applicableforNBandWB(AMRandAMRWBcoding,CDMAtestsongoing)
resultscomputedonapersentencebasisandaveragedoverallsamples
ETSI2012.Allrightsreserved
Chapter3
Fundamentalset
Fundamentalsetuptocollect
recordings
ETSI2012.Allrightsreserved
SetupbasedonETSIES
ETSIES202396
2023961
Objectivemodelsneed3inputs:clean,noisyandprocessed
clean:atourdisposal,signaltransmittedtotheHATSsmouth
noisy:mixtureofcleanandbackgroundnoise
useofdedicatedloudspeakerarrangementstatedinES2023961toreproduceinalab
environmentrealisticbackgroundnoiseconditions
noisysignalpickedupbyareferencemicrophonepositionedclosetotheprimary
microphoneofthemobileterminal
processed:mobileoutputofthenoisysignaltransmittedbythemobilephone
processed: mobile output of the noisy signal transmitted by the mobile phone
requiresanetworksimulator(controllednetwork)
Loudspeaker
rear right
rearright
Networksimulator
Loudspeaker
rearleft
Loudspeaker
frontleft
Reference
microphone
Subwoofer
Backgroundnoise
simulator
ETSI2012.Allrightsreserved
Loudspeaker
frontright
UpdateofEG202396
UpdateofEG
2023963
3
ES
ES202396
2023963
Fundamentalcalibrationandequalizationproceduresofthe
d
l lb
d
l
d
f h
backgroundnoisesimulationsystem
impactoftheloudspeakerspositioning:cornersshallbeavoided
impactoftheroom:noinfluenceonlyandonlyifproperequalization
useofstaticdelaysinsteadofautomaticdelays
nosharpfiltering
startequalizationfrom50Hztocoverwiththepresenceofwidebandterminals
start equalization from 50 Hz to cover with the presence of wideband terminals
interlabsrepeatablemeasuresonlyifpositioning,calibrationandequalizationstepsare
doneproperlyontheentirefrequencyband
Differenttypesofrealbackgroundnoiseavailable
insidecarnoise,insidebusnoise,insidetrainnoise,
inside car noise, inside bus noise, inside train noise,
publicplacesnoise,
workplacenoise,
8noisesselectedby3GPPandSA4forNoiseReductionevaluation
pub,road,crossroads,train,fullsizecar,cafeteria,mensa,office
9
ETSI2012.Allrightsreserved
Chapter4
Analysisofmeasurementsonmodern
mobilephones:selectedexamples
10
ETSI2012.Allrightsreserved
DescriptionofOrangedata(Widebandonly)
Noisereduction:useof6differentWideBandmobileterminals
from6differentmobileproviders(D1,D2,D6)
variousacousticdesigns
someequippedwithmonomicrophoneandotherswithdualmicrophones
setupinaccordancewithES2023961
useoftheACQUAandtheHAEBGNsystemsfromHEADacoustics
network simulator CMU200
networksimulatorCMU
200fromRohde&Schwarz
from Rohde&Schwarz
speechsamples:useof6talkers(3males,3females),2sentencespertalker
backgroundnoiseconditionslimitedtothe8noisesselectedby3GPPandGSMA
P.835subjectivetests
performedbyandatOrangeLabswithnaivelisteners(24votespersample)
twoP.835tests:
onewithroad,train,fullsizecar,cafeteriaandofficenoises providedfor3QUESTretraining
onewithpub,mensaandcrossroadsnoises providedforvalidationoftheretrained3QUEST
11
ETSI2012.Allrightsreserved
Comparison ofobjectiveandsubjectiveresults
ofobjectiveandsubjectiveresults
OrangeWBdata
Orange WB data
SMOS
5
4,5
MOSscore
4
3,5
EG2023963
P.835
2,5
TS103106
2
1,5
D2
D3
D4
D5
D6
NMOS
5
4,5
Improvedrobustnesstonoise
Improved robustness to noise
MOSscore
4
3,5
EG2023963
P.835
2,5
TS103106
2
1,5
D1
D2
D3
D4
D5
D6
GMOS
4,5
4
MOSscore
35
3,5
EG2023963
P.835
2,5
TS103106
2
1,5
1
D1
ETSI2012.Allrightsreserved
D2
D3
D4
D5
D6
reductionsystems
withEG2023963,D1
and 6 were badly
andD6werebadly
estimated(upto1.0MOS
difference)
12
Averagedover8noise
conditions,goodpredictions
conditions,
good predictions
forall3dimensions(S,N,and
GMOS)withTS103106
1
D1
TS103106exhibitsmuch
betterperformancethan
EG2023963
Comparisonofobjectiveresults
Comparisonofobjectiveresults
Orange WB data
OrangeWBdata
A l i f bj i d f
Analysisofobjectivedatafrom52WBmobileterminals
52 WB
bil
i l
EG2023963ismorepessimisticonSMOS:TS103106,closertohumanperception,smoothes
performance (3.5 SSMOS
performance(3.5
MOS
4.1)
EG2023963providesNMOShigherthanTS103106(0.5MOS)
SMOS
4,5
4
3,5
3
EG2023963
2,5
TS103106
2
1,5
1
A B
G H
L M N O P Q R
T U V W X
Z A1 A2 C1 D1 E1 F1 G1 H1 I1 J1 K1 L1 M1 N1 O1 P1 Q1 R1 S1 T1 U1 V1 W1 X1 Y1 Z1
NMOS
OS
5
4,5
4
3,5
EG2023963
TS103106
2,5
2
1,5
1
A B
13
ETSI2012.Allrightsreserved
G H
L M N O P Q R
T U V W X
Z A1 A2 C1 D1 E1 F1 G1 H1 I1 J1 K1 L1 M1 N1 O1 P1 Q1 R1 S1 T1 U1 V1 W1 X1 Y1 Z1
Comparisonofobjectiveresults
Comparisonofobjectiveresults
HEAD acoustics WB data
HEADacousticsWBdata
A l i of16WBhandheld
Analysis
f 16 WB h d h ld HFTmobileterminals:Cafe
HFT
bil t
i l C f Noise63dB(A)
N i 63dB (A)
Wide rangeofSMOSwith EG2023963,typically lower than TS103106based tests
Wide rangeofNMOSwith
range of N MOS with EG2023963,typically
EG 202 396 3 typically lower than TS103106based
TS 103 106 based tests
5.0
G-MOS Range
G-MOS / Caf
N-MOS / Caf
S-MOS / Caf
S-MOS min
N-MOS min
S-MOS avg
N-MOS avg
GMOS avg
4.5
4.0
3.5
3.0
2.5
2.0
1.5
-D
ev
1
1
-D
e
M
v2
2
-D
ev
M
1
2
-D
e
M
v2
2
-D
ev
M
3
2
-D
ev
M
4
2
-D
e
M
v5
3
-D
ev
M
1
3
-D
ev
M
2
3
-D
e
M
v3
4
-D
ev
M
1
4
-D
ev
M
2
5
-D
e
M
v1
5
-D
ev
M
2
6
-D
e
M
v1
6
-D
ev
2
1.0
EG 202 306-3
14
ETSI2012.Allrightsreserved
---
---
TS 103 106
Comparisonofobjectiveresults
Comparisonofobjectiveresults
HEAD acoustics WB data
HEADacousticsWBdata
A l i f 16 WB h d h ld HFT
Analysisof16WBhandheldHFTmobileterminals:CarNoise69dB(A)
bil t
i l C N i 69 dB(A)
WiderrangeofSMOSwithEG2023963,typicallylowerthanTS103106basedtests
WiderrangeofNMOSwithEG2023963,typicallylowerthanTS103106basedtests
Wide rrange of N MOS with EG 202 396 3 typically lower than TS 103 106 based tests
HigherNMOSthanSMOSforsomedeviceswithEG2023963,SMOSalwayshigherwithTS103106
G-MOS Range
G-MOS / Car
N-MOS / Car
S-MOS / Car
S-MOS min
N-MOS min
S MOS avg
S-MOS
N-MOS avg
GMOS avg
5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5
-D
ev
1
1
-D
ev
M
2
2
-D
e
M
v1
2
-D
ev
M
2
2
-D
e
M
v3
2
-D
ev
M
4
2
-D
ev
M
5
3
-D
e
M
v1
3
-D
ev
M
2
3
-D
e
M
v3
4
-D
ev
M
1
4
-D
ev
M
2
5
-D
e
M
v1
5
-D
ev
M
2
6
-D
e
M
v1
6
-D
ev
2
1.0
EG 202 306-3
15
ETSI2012.Allrightsreserved
---
---
TS 103 106
Comparisonofobjectiveresults
Comparisonofobjectiveresults
HEAD acoustics WB data
HEADacousticsWBdata
A l i f 16 WB h d h ld HFT
Analysisof16WBhandheldHFTmobileterminals:RoadNoise74dB(A)
bil t
i l R d N i 74 dB(A)
Muchlowervaluesthanfortheothernoises
SimilarrangeofSMOSbutlowerforEG2023963thanforTS103106basedtests
Similar range of S MOS but lower for EG 202 396 3 than for TS 103 106 based tests
SimilarrangeofNMOSwithEG2023963,slightlylowerthanTS103106basedtests
G-MOS Range
G-MOS / Road
N-MOS / Road
S-MOS / Road
S-MOS min
N-MOS min
S MOS avg
S-MOS
N-MOS avg
GMOS avg
5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5
-D
ev
1
1
-D
ev
M
2
2
-D
e
M
v1
2
-D
ev
M
2
2
-D
e
M
v3
2
-D
ev
M
4
2
-D
ev
M
5
3
-D
e
M
v1
3
-D
ev
M
2
3
-D
e
M
v3
4
-D
ev
M
1
4
-D
e
M
v2
5
-D
ev
M
1
5
-D
ev
M
2
6
-D
e
M
v1
6
-D
ev
2
1.0
EG 202 306-3
16
ETSI2012.Allrightsreserved
---
---
TS 103 106
Conclusions
NewETSIstandardTS103106availableforevaluationof
mobilephonesinbackgroundnoise
mobile phones in background noise
RevisedETSIstandardES2023961availablewithbetter
definition of background noise simulation in wideband
definitionofbackgroundnoisesimulationinwideband
IngeneralSMOSestimationbasedonTS103106more
optimistic than with EG 202 396 3 less variation/less
optimisticthanwithEG2023963,lessvariation/less
discriminativebutbetteradaptedtoqualityperceptionof
mobile user
mobileuser
IngeneralNMOSestimationbasedonTS103106more
pessimistic than with EG 202 396 3 but very smilar in trend
pessimisticthanwithEG2023963butverysmilarintrend
17
ETSI2012.Allrightsreserved